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PARENTS’ VIEWS ON THE USE OF AI-BASED CHATBOTS SUCH AS CHATGPT IN 
HIGH SCHOOL (STEM) EDUCATION 


Markus Sebastian Feser 
Leibniz Institute for Science and Mathematics Education, Germany 


Introduction 


As digital technology becomes increasingly available in education, educational researchers around the 
globe increasingly have to confront how it may affect school students’ academic success. With the emergence of 
generative Artificial Intelligence (Al) and Al-based chatbots—especially ChatGPT—this issue has attracted even 
more attention (Lo, 2023), particularly in Science, Technology, Engineering, and Mathematics (STEM) education 
research (e.g., Bitzenbauer, 2023; Cooper, 2023; Nasution, 2023; Vasconcelos & dos Santos, 2023; Zhai, 2023). For 
example, within the last 12 months, educational researchers around the globe have investigated the academic 
and media discourse on the use of Al-based chatbots in educational settings (e.g., Fowler et al. 2023; Jarrah et al., 
2023), assessed the quality of responses generated by such chatbots (e.g., Cooper, 2023; de Winter, 2023; Frieder 
et al., 2023; Nasution, 2023), and explored whether and to which extent classroom activities that incorporate the 
use of Al-based chatbots may benefit teaching and learning processes (e.g., Baidoo-Anu & Owusu Ansah, 2023; 
Bitzenbauer, 2023; Imran & Almusharraf, 2023; Vasconcelos & dos Santos, 2023; Zhai, 2023). However, aside from 
the results of some rather holistic opinion polls (e.g., Hartley, 2023; Vodafone Foundation Germany, 2023), hardly 
any research has assessed parents’ views on the use of Al-based chatbots in high school education, especially when 
it comes to STEM education. 

Numerous studies have provided evidence that parents have a decisive impact—presumably even the most 
decisive impact—on their children’s educational outcomes and influence children’s academic performance and 
attitudes through their support, beliefs, and expectations (e.g., Ceka & Murati, 2016; Fan & Chen, 2001; Hill & Tyson, 
2009; Ma, 2001). Notably, parents’ impact also extends to the use of digital technology in high school education, 
since several studies have verified that parents crucially shape their children’s perceptions of digital technology 
(e.g., Kong et al., 2019; Kotrla Topic¢ et al., 2020; Ortiz et al., 2011). This is especially the case when parents themselves 
hold positive views regarding digital educational technology; their children also tend to view digital educational 
technology in the same way (Ortiz et al., 2011). This, in turn, reinforces the need to understand parents as impor- 
tant stakeholders in educational systems when considering the expansion and ongoing implementation of digital 
technology in high school education (Maxwell et al., 2021). Therefore, the aim of the present editorial is to provide 
some initial insights into parents’ views on the use of Al-based chatbots, such as ChatGPT, in high school (STEM) 
education and to offer some suggestions for forthcoming research. 
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Parents’ Views on the Use of Al-Based Chatbots Such as ChatGPT in High School (STEM) Education 


To this end, | conducted an exploratory survey with a convenience sample of parents from around the globe. This 
survey was an anonymous and voluntary online questionnaire administered via SurveySwap (www.surveyswap.io). 
The sample | surveyed consisted of 73 parents, and the survey was conducted between May and September 2023. 
Female participants constituted the majority (65.8%), followed by males (31.5%), and a small percentage identi- 
fied as neither female nor male (2.7%). Geographically, the surveyed parents predominantly hailed from Europe 
(76.7%), with fewer from Central, North, and South America (12.3%), Asia (6.9%), Africa (2.7%), and Oceania (1.4%). 
To assess participants’ views on the use of Al-based chatbots in high school (STEM) education, | implemented 
three scales based on adapted versions of items developed by Maxwell et al. (2021) (response format: 1 = strongly 
disagree to 5 = strongly agree). These three scales (see Figure 1) captured the extent to which participants held a 
positive stance toward the use of Al-based chatbots such as ChatGPT in high school education in general (M = 3.05, 


SD = 0.80, O.,. nach = 0-82, gib = 0.89, number of items = 5), the extent to which they held a negative stance toward 
such use of Al-based chatbots (M = 2.72, SD = 1.04, a...) = -79, gib = 0.80, number of items = 3), and participants’ 
stance toward the use of such chatbots in high school STEM education (M = 3.55, SD=0.83, 4... = 68, glb = 0.70, 


number of items = 3). Moreover, using single-item measures, | asked participants how often they used Al-based 
chatbots such as ChatGPT themselves (response format: 1 = never, 2 = once, 3 = occasionally, 4 = frequently; 
M = 2.70, SD = 1.05) and how they would rate their own knowledge about Al-based chatbots (response format: 
1 =very poor to 10 = excellent; M = 5.27, SD = 2.39). 


Figure 1 
Participants’ Response Behavior Across the Three Scales for Assessing Parents’ Views on the Use of ChatGPT in High School 
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Figure 1 summarizes participants’ response behaviors across the three scales used to assess parents’ views on 
the use of Al-based chatbots such as ChatGPT in high school education. To begin with, the participants’ responses 
to the items that explicitly addressed STEM education revealed a consistently positive stance toward the use of 
Al-based chatbots in this regard. A clear majority of the participants believed that the use of such chatbots may 
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enhance high school students’ learning, especially within STEM classes (54%). Furthermore, they (strongly) agreed 
that the use of such chatbots should be part of the STEM curriculum in high school (65%), and they (strongly) 
agreed that it should be the responsibility of STEM teachers to teach about Al-based chatbots such as ChatGPT 
in high school (61%). 

However, the participants’ response behaviors across the two scales that addressed the use of Al-based chat- 
bots, such as ChatGPT in high school education, in general, revealed a rather heterogeneous pattern. On the one 
hand, a clear majority of the participants did not believe that the use of such chatbots in high school education 
may negatively affect students’ learning (67%), supported the use of such chatbots in high school education in 
general (60%)—but at the same time thought that high school teachers should not prioritize them over textbooks 
(73%)—and considered Al-based chatbots to be a valuable tool for high school teaching and learning (62%). On 
the other hand, participants’ responses to four items did not indicate a clear majority or consensus in their views on 
the use of Al-based chatbots such as ChatGPT in high school education. Instead, participants’ views seemed divided 
as to whether the use of such chatbots may decrease the quality of high school teaching (39% [strongly] agree; 
40% [strongly] disagree), whether the use of such chatbots should be adopted in all subjects taught in high school 
(34% [strongly] agree; 41% [strongly] disagree), whether high school teachers should reinforce their students’ use 
of such chatbots (46% [strongly] agree; 25% [strongly] disagree), and finally, whether the use of Al-based chatbots 
in high school education should be prohibited (35% [strongly] agree; 47% [strongly] disagree). 


Table 1 
Pearson/Point-Biserial Correlations with Participants’ Background Information 


Pearson/Point-Biserial Correlation 


Variable 
Positive Negative STEM 
Gender identification (dichotomized: 1 = female; 2 = non-female) -.04 01 08 
Geographical origin (dichotomized: 1 = European; 2 = non-European) -.25* 27* -.16 
Own use of Al-based chatbots such as ChatGPT (self-evaluation) A7 -.09 242 
Own knowledge of Al-based chatbots such as ChatGPT (self-evaluation) .23* -.20° 38*** 


Note. Positive = positive stance on the use of ChatGPT in high school education; Negative = negative stance on the use of 
ChatGPT in high school education; STEM = stance on the use of ChatGPT in STEM education in high school; ° p < .10;* p < .05, 
** p< .01; *** p < .001. 


In addition to the above, a correlation analysis based on the three scales for assessing parents’ views on the 
use of Al-based chatbots such as ChatGPT in high school education and participants’ background information 
yielded some remarkable patterns (see Table 1). Specifically, this analysis revealed significant correlations between 
participants’ geographical origin (European/non-European) and their views on the use of Al-based chatbots such 
as ChatGPT in high school education in general, as well as between their views on the use of such chatbots in high 
school (STEM) education and their self-evaluated knowledge of Al-based chatbots. Regarding participants’ gender 
identification and their own use of Al-based chatbots, (narrowly) no significant correlations emerged. 


Outlook 


To summarize, the survey | conducted revealed that parents generally hold a positive view of integrating Al- 
based chatbots such as ChatGPT into high school STEM education, with most believing it enhances learning and 
should be part of the STEM curriculum. However, the participants’ views diverged on the broader use of such chatbots 
in high school education, with mixed stances on their impact and adoption in all high school subjects. Moreover, 
a correlation analysis revealed links between participants’ views on the use of Al-based chatbots in high school 
education and their geographical origin, as well as their self-evaluated knowledge about the use of such chatbots. 

Given that the survey | presented in this editorial was exploratory and the convenience sample was rather small, 
the results are clearly not generalizable, especially since, for example, parents from Europe and female parents were 
heavily over-represented. Instead, the results of this survey should be seen as initial insights into parents’ views on 
the use of ChatGPT in high school (STEM) education, which may motivate future educational research. To suggest 
only a few examples, based on these initial insights, future research could conduct investigations on the following: 
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e¢ Whether the response patterns and correlations detailed above may also occur or substantially differ 
within larger and more robust samples of parents, 

e Whether additional background information (e.g., parents’ vocational background or level of educa- 
tion) also correlates with parents’ views on the use of Al-based chatbots such as ChatGPT in high school 
education, 

¢ Potential causes of the response patterns and correlations detailed above, if they can be replicated 
within future studies, 

e Whether parents’ views on the use of Al-based chatbots such as ChatGPT in high school (STEM) education 
may change over time as generative Al and Al-based chatbots become more and more commonplace, or 

e What specific concerns and expectations do parents have regarding the use of ChatGPT in high school 
(STEM) education (e.g., via in-depth interviews), especially when it comes to the role of STEM teachers 
in this regard? 

To conclude, the use and potential benefits of Al-based chatbots such as ChatGPT within high school (STEM) 
education is a novel area of educational research with a broad variety of research gaps. To the best of my knowl- 
edge, this is especially true when it comes to parents’ views in this regard. Therefore, it is plausible to assume that 
delving into an in-depth inquiry of parents’ attitudes, perspectives, and concerns about the use of such chatbots 
in educational settings would substantially contribute to bridging some crucial voids in our understanding of 
the overall impact of these digital technologies on high school (STEM) education and thus may be a particularly 
promising area for future research. 
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Abstract. Online teaching has become 

an imperative approach in today’s society. 
However, as an essential approach, using 
live streaming to teach students in small 
groups, particularly rural primary school 
students, has not been extensively studied. 
To address this gap, an inquiry-based 
model, predict-do/observe-quiz/discuss- 
explain-transfer (P-D/O-Q/D-E-T), was 
adopted for live streaming with action, and 
its performance was analysed. Reflection 
by teachers can lead them to a deeper 
understanding to capture the profound 
impact of an educational program. In the 
present study, eight teachers who had 
experience assisting rural students in this 
experiment were invited to rate their points 
of view on immersion, social interactivity, 
humanness, and value perception. 
Examining the consistency of teachers’ 
viewpoints using the hermeneutic method, 
the results showed that they highly 
supported viewing these four constructs 
using live streaming to conduct inquiry- 
based science learning with action. As 
expected, using live streaming to deliver 
teaching with the P-D/O-Q/D-E-T approach 
can enrich other online science teaching. 
Keywords: educational program, inquiry- 
based science learning, live-streaming, 
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LIVE-STREAMING 
PERFORMANCE IN INQUIRY- 
BASED SCIENCE LEARNING 
WITH ACTION: TEACHERS’ 
PERSPECTIVES 


Jon-Chao Hong, 
Huei-Ying Ho, 
Ming-Chou Liu, 
Kai-Hsin Tai 


Introduction 


The rise of live streaming has highlighted the importance of under- 
standing how the embedded functions of learning platforms, known as 
affordances, shape students’ experiences and impact teachers’ attitudes to- 
ward students’ needs. In particular, live streaming platforms, where students 
can watch real-time and live videos delivered by streamers, have become 
one of the most important platforms for use in distance education today. 
However, how live-streaming functions embedded in science learning can 
be leveraged for inquiry-based learning has not been extensively studied. 
Thus, the present study used a live-streaming approach to teach science to 
rural students. Live streaming effectively enhances communication and en- 
tertainment promotion through the Internet's intuitive design, rich content, 
robust interactivity, unrestricted location, and audience segmentation (Yu et 
al., 2018). Streaming feedback allows the streamer to adjust the program's 
content to meet the user's needs (Bilal et al., 2018). In the past, live stream- 
ing was mostly limited to electronic entertainment media. However, with 
the increasing popularity of broadband networks and the improvement of 
computing power, live streaming is now being used for educational purposes 
through the Internet (Chen & Lin, 2018; Payne et al., 2017). For example, real- 
time surgery conditions can be transmitted via live streaming, and most of 
the students who watched the live stream showed a significant subjective 
increase in knowledge after watching the live stream. The findings suggested 
that live-streamed surgeries should be offered as a permanent component of 
medical teaching (van Bonn et al., 2022). However, few studies have examined 
inquiry-based science learning, in particular, following Dewey's “learning by 
doing” educational pedagogy (DuFour et al., 2016). Therefore, the present 
study developed an inquiry-based science learning with an action approach, 
entitled predict-do/observe-quiz/discuss-explain-transfer (P-D/O-Q/D-E-T), 
for rural primary school students to learn science. To solve the problem of 
the lack of science teachers available to teach in rural areas of Taiwan, a live- 
streaming platform was implemented to conduct four P-D/O-Q/D-E-T lessons. 
To explore students’ experience of live streaming to practice inquiry-based 
learning, the on-site assisting teachers observed and reported the students’ 
performance effectiveness. 
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According to the socio-technical system theory, the essential factor for optimizing output is the synergy 
between the technical component (technology) and the social component (interpersonal interactions) (Kong et 
al., 2019). The unique advantages are the real-time interactivity, visualization, and personalized services fostered 
by live streaming (Zhang et al., 2022). In line with this, P-D/O-Q/D-E-T inquiry-based science learning with action 
was designed for rural primary school students, and how teachers perceived their students’ involvement in the 
socio-technical system was reported in this study. The current research posited that users’ inclination towards live 
streaming is contingent upon their perception of the matching degree of task difficulty and their capability (Zhao 
et al., 2018). Confronting challenges, they can perceive the positive aspects of the task, view the challenging stress 
as a driving force, and consider obstacles as pressure (Gonzalez-Morales & Neves, 2015). This study aimed to bridge 
the present gap by examining the driving factors behind the intention to broadcast on educational live-streaming 
platforms. Unlike the development in the field of practice, live streaming is still not receiving sufficient attention in 
the academic field (Li et al., 2021). This led to the research question: What is the role of perceived value, perceived 
humanness, social interactivity, and immersion in users’ participation in the live-streaming process? The results of 
this study can provide some evidence for helping science teachers teach science to students in rural areas around 
the world. 


Theoretical Background 
Performance Measurement 


User experience involves the thoughts and feelings of using or anticipating online services. This is influenced 
by the personal-internal state, the system's characteristics, and the interactional context (Tham & Grace, 2020). To 
understand this experience more deeply, one can use a performance measurement that regularly measures the 
outcomes and efficiency of services or programs (Hatry, 2006). Gadamer (2008) defined “deeper understanding” 
as the capacity to question the subject matter and formulate further questions (p. xxii). This means that it is im- 
portant to consider the value of interpretation when formulating the meaning of learning offered by participants. 
This involves listening, observing, testing, and reflecting from different angles (Du Plessis & McDonagh, 2021). 

One way to evaluate user performance is to assess how effectively users are able to accomplish tasks within 
a given system (Habibi & Chattopadhyay, 2021). In order to retain and engage users, it is crucial to facilitate a posi- 
tive and interactive relationship between viewers and streamers (Scheibe et al., 2016). Teachers’ interpretations 
of students’ attitudes, behaviour, and cognition are closely linked to the way their students learn. By reflecting on 
their students’ learning process, teachers can deeply understand an educational program's profound impact (Du 
Plessis & McDonagh, 2021). Gadamer’s theory (1975) enabled the merging and rationalization of perceptions and 
experiences across different participants in diverse settings, prompting Mayer (2014) to propose the use of the 
Cognitive Affective Theory of Multimedia Learning (CATML) for investigating how affective and cognitive factors 
could impact learning positively or negatively. Thus, the present study looked at the performance measurement 
from the teachers’ point of view in their students’ cognitive and affective states in inquiry-based science learning 
with action (IBSLA) through live streaming to rationalize beyond the science concept learning. 


Immersion 


When oneis highly focused during an activity, one may experience a sense of success and inner peace, as well 
as feelings of joy and wonder. This feeling aligns with the idea of being fully immersed or present in online learning 
activities. Studies have found that the greater the challenge, the greater the sense of immersion (Shernoff, 2010). 
To achieve this level of engagement, instructional designers should develop learning activities that align with the 
core element of immersion, which will spark students’ curiosity and capture their interest (Petersen et al., 2022). The 
use of live streaming can be beneficial in promoting distributed cognition by providing a platform for engagement 
between participants, instructors, and objects in the context of IBSLA. To create an effective learning environment, 
it is important to establish a framework that addresses students’ learning needs and instructional objectives while 
also holding personal significance and applicability for the students. This approach can help students become more 
engaged in their science coursework and feel a sense of immersion in the learning experience. In the world of 
online education, students and teachers use digital avatars to show whether they are actively participating or not 
(Hong et al., 2022). The state of immersion emerges from the fidelity of representations and the level of interaction, 
making it not inherently singular (Petersen et al., 2022). However, it is worth acknowledging that these definitions 


10 


https://doi.org/10.33225/jbse/24.23.09 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


LIVE-STREAMING PERFORMANCE IN INQUIRY-BASED SCIENCE LEARNING WITH ACTION: ISSN 1648-3898 /Print/ 
TEACHERS’ PERSPECTIVES 
age ISSN 2538-7138 soniine/ 


suggest the importance of recognizing immersion, which has also been defined in educational contexts (e.g., Yeo 
et al., 2020). Thus, how IBSLA delineates context and meaning through a live-streaming environment that facilitates 
students’ immersion in learning science for the successful completion of the live-streaming learning activity was 
explored in this study. 


Perceived Social Interactivity 


Described as a’sense of being there and interacting with another; social interactivity implies that the emotional 
bonds formed in mediated environments directly impact how people communicate and behave with others (Biocca 
et al., 2003). More specifically, social interactivity refers to the individual's ability to utilize internal socio-networking 
tools to connect with peers and utilize system features to engage in collaborative activities (Paramita et al., 2021). 
Empirical evidence suggests that offline socio-interactions significantly shape the dynamics of online experiences 
(Webb, 2001). Engaging in socio-interaction enables individuals to draw insights from socializing, participating, 
and receiving input from the virtual community (Hollebeek & Macky, 2019). In other words, participants must en- 
gage diverse cognitive functions through social interactivity to problem solve and attain their learning objectives 
(McCreery et al., 2012). 

In our live streaming case, the streamer guided participants following a social framework and a set of inter- 
active rules (e.g., 1 minute for thinking, 2 minutes for discussing with group members, and then presenting their 
answers to the streamer) and instructed students to watch presentation files. Moreover, the streamer proposed 
questions for students to predict and discuss with other participants as the social interactivity in the live-streaming 
environment. Considering the learning goals may be hindered since the learners are in a technology-driven learn- 
ing environment (Petersen et al., 2022), how participants perceived social interactivity is an important component 
affecting their engagement in the live-streaming learning process. Thus, the way participants perceived social 
interactivity was explored in this study. 


Perceived Humanness 


According to Lortie and Guitton (2011), the reciprocity of exchanges between humans and agents is a crucial 
factor in perceived humanness. Regarding conversation structure, if an agent gives a question related to a previous 
question or response (i.e., it is easy to figure out the answer from the context presented), it will positively influence 
the perceived humanness (Hill et al., 2015). In contrast, when a question does not consider the context provided 
in the previous content, it is expected to have a negative impact on perceived humanness (de Kleijn et al., 2019). 
Therefore, asking immediate questions related to the ongoing discussion is essential. 

Collaborating and discussing tasks online can be done through live streaming. To promote IBSLA, it is im- 
portant to have easily accessible tools and resources available. This study provided participants with various tools 
and objects (such as iPads and experimental materials), enabling them to take advantage of the benefits of active 
learning through live streaming. Using shared tools (such as Zoom, Google Meet, and Teams), students were able 
to engage in activities that required active learning and built an interactive model of IBSLA. The experimental 
materials could be easily recognized and obtained by all participants either through delivery from researchers 
or preparation by local teachers. That is, this study offered an interactive platform that helped establish a shared 
cognitive foundation for activities among both students and teachers. 

Moreover, Lortie and Guitton (2011) found a relationship between the number of questions asked and the 
agent's perceived humanness. Beyond the visual element, interactive structure, timing, and content that make up 
the design of a remote human, voice is another prominent characteristic of particular interest (Heidig & Clarebout, 
2011).When participants in a learning environment have access to good visuals and voice, they can actively engage 
in the learning activity, which is crucial in developing their knowledge. This study investigated how participants 
perceived humanness and how it affected their engagement in live-streaming learning. 


Perceived Value 


According to the Situated Expectancy-Value Theory (SEVT; Eccles & Wigfield, 2020), teachers’ attitudes towards 
the subjects they teach and their perceived value for their students are crucial factors in their teaching. Values sig- 
nificantly impact individuals’ cognition and coping mechanisms that affect their goals and emotions (Hernandez- 
Ortega et al., 2017). When individuals believe that acquisition and involvement are important for their personal 
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happiness and identity, they adopt effective communication strategies to manage their intentions (Zhou et al., 
2019). Live-streaming platforms offer several advantages, such as a high degree of interactivity, a strong sense of 
involvement, privacy, and novelty seeking (Yu et al., 2018). Perceived value significantly impacts the users’ continued 
use of streaming services, followed by perceived enjoyment (Singh et al., 2021). By understanding the cognitive 
appraisal process, one can assess how live streaming affects individuals’ learning and performance (Clarke, 2012). 
Accordingly, this study examined how participants perceived the learning value of using live streaming. 


Research Methodology 
Background 


Husserl (1990) presented a method of descriptive phenomenology that aimed to reveal the fundamental 
essence of phenomena relevant to questions of knowledge. This involved setting aside existing knowledge and 
preconceptions about the phenomenon to uncover its core nature. Heidegger (1962), on the other hand, intro- 
duced a hermeneutic or existential phenomenology approach that emphasized understanding the essence of 
‘being’ through interpretive sciences. This approach focused on exploring questions about existence to reveal 
insights and significance. Gadamer (2008) proposed that a profound understanding was inherent to ‘being’ and 
advocated using the Hermeneutic method. This method supported phenomenological observation by providing 
an interpretive approach (Betensky, 1995) to reflect on the meanings of phenomena as they manifested in obser- 
vational experiences (Santiago et al., 2020). 

The foundational philosophical principles of phenomenology proposed by van Manen (2014) encompassed 
two main approaches: descriptive phenomenology and interpretive (hermeneutic) phenomenology, both focused 
on understanding the human experience of phenomena. According to van Manen (1990), researchers engaged 
participants in questioning to gain profound insights into these phenomena as part of a methodology rooted in 
human science. Phenomenological reflection involves identifying essential themes that encapsulate the funda- 
mental meanings of these phenomena. This process involves researchers delving into thematic analysis, selecting 
pertinent thematic statements, capturing thematic expressions, and contemplating the significance of observed 
experiences through an observational checklist. The current study developed a specialized checklist for educa- 
tors to document their phenomenological observations of evaluating student performance within the context of 
P-D/O-Q/D-E-T lessons for science-based inquiry learning. 


Data Collection 


The research of van Manen (2014) suggested that the human science method provides researchers with 
clear guidelines for collecting and analysing data. One of the main concerns surrounding learning is the inequal- 
ity among students. Teachers have observed that some students have limited access to online learning materials 
(Kim & Asbury, 2020). Additionally, during synchronous teaching, the restriction of learning-related interaction has 
made it difficult for teachers to address their students’ needs due to the limited feedback they receive (Haser et al., 
2022). This study used observational field notes and comprehension checks as primary data sources. 

In this study, five physics units, such as surface tension, buoyancy, and so on, were conducted with live stream- 
ing in the fall semester of 2022. Every unit was implemented in the morning before regular class for 30 minutes. 
Eight rural schools joined this study, with one teacher participating in each school. Finally, six teachers returned 
their checklists for statistical analysis. 


Instrument 


Five domain experts had to explicitly state the assessment constructs and items based on the existing study 
of the checklists. For the systematic review, each domain expert analysed constructs and assessment items found 
to be potentially relevant to this study. Afterwards, an expert panel meeting was conducted to gain agreement 
on the meaning of each item and construct in the previous template. Consequently, the checklist consisted of 
four constructs with six items each. Moreover, Gadamer (1975) described lived experience as a “significant whole” 
made up of several parts or clusters of meaning, a framing that underlines the need to “understand the whole in 
terms of the detail and the detail in terms of the whole” (p. 258). Thus, perceived humanness and perceived social 
interactivity were adapted from Fernandes and Oliveira (2021), and immersion and perceived value were adapted 
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from Li et al. (2021). Thus, questionnaire items were designed as listed below. 


Immersion 
1) My students are immersed in the live-streaming environment. 
2) My students felt that the live-streaming session ended faster than they expected. 


3) My students did not feel frustrated with the length of speaking in the live-stream sessions. 
4) My students lost outside awareness when they joined the live-streaming course. 

5) My students were so excited when they interacted with the live-streaming platform. 

6) My students’ eyes were blinking when they were in the live-streaming sessions. 


Perceived Social Interactivity 

1) I find that my students are pleasant to interact with during discussion sessions. 

2) | feel that the live streamer could understand my students’ replies right away. 

3) When interacting with the live streamer, my students could process the content easily. 

4) When watching a live stream, my students could exchange and share opinions with the streamer. 

5) When my students were watching a live stream, the streamer provided sufficient opportunities to 
respond and ask questions. 

6) When watching a live stream, my students could exchange and share opinions with other school 
students easily. 


Perceived Humanness 
1) Sometimes my students felt that the live streamer seemed to really be in the classroom. 
2) When my students were watching a live stream, the streamer knew they were close to him or her. 


3) When watching a live stream, my students felt close to the streamer. 

4) When my students were watching a live stream, the platform processed their comments very quickly. 
5) When my students were watching a live stream, seeing comments sent by other viewers was very fast. 
6) When my students were watching a live stream, when they voted, they could see the result very fast. 


Perceived Value 

1) I think the live-stream program with inquiry learning is useful for students to learn science. 

2) | think the live-stream program with inquiry learning can help students develop an inquiry attitude 
towards science. 

3) think students can get used to the P-D/O-Q/D-E-T process after several practices. 

4) | think students can retain those science concepts learned via the live-stream program longer than 
those learned in a normal class. 

5) think my students can transfer their science learned from live-stream programs to daily events. 

6) | think my students are interested in using live streaming to learn science. 


Data Analysis 


In behavioural sciences research, it is crucial to ensure reliability and agreement. Reliability refers to how well 
a scale distinguishes between items when categorized, while agreement measures how close two assessments of 
the same items are (Gearhart et al., 2013). Good reliability is important for ensuring the validity of a measurement 
scale, particularly when assessing the agreement of variables between observers and the attenuation effect (Lee, 
1997). The Kendall's W statistic, also knownas the coefficient of concordance, is anon-parametric measure utilized to 
evaluate the level of agreement among various raters. It ranges from 0 to 1, where 0 indicates no agreement among 
raters and 1 represents perfect agreement. The statistic is calculated on an interval or ordinal scale (Howell, 2002). 
Accordingly, Kendall’s W statistic can be used to assess the quality of a live stream for science inquiry-based learning. 

Kendall's coefficient of concordance (W) is employed to gauge the level of concordance between random 
variables and their order statistics (Fuchs & Schmidt, 2021). In this study, the interrater reliability (McHugh, 2012) 
of the usability classification system was evaluated. Following institutional review board approval, six raters par- 
ticipated in an experiment centred around inquiry-based science learning facilitated through live streaming. The 
raters observed students’ management of achievement emotions and their associated control value during this 
experiment. The calculation and reporting of interrater reliability were conducted using Kendall’s W. The formula 
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of Kendall’s W is as follows (Kendall & Gibbons, 1990): 


sa 125 ' 
See eq. 
k2(N3 — N) : 


N 
s=) (=F cs eq. 2 
i=1 


R: Rating scale (3 ranks in this study). 

N: Assessing items and constructs (4 constructs with 6 items in each construct). 
K: Total rating members (6 pedagogical experts in this study). 

S: The sum of squared deviations. 


If Wis 0, the raters have no overall agreement trend, and their responses may be considered random. Inter- 
mediate values of W indicate a greater or lesser degree of unanimity among the raters, so S will have the maximum 
value when the raters are entirely consistent. The harmony coefficient is the ratio of the actual obtained S to its 
maximum possible value, as0 < W< 1. 


Research Results 


Drawing on the literacy review, for this study, four variables were designed to measure the quality of live 
streaming for students to learn science, namely immersion, perceived social interactivity, perceived humanness, 
and perceived value for students to learn. Pedagogical experts (who participated as observers and facilitators 
during the live streaming) were invited to rate on a 3-point ranking scale (i.e., 1 for disagree, 2 for neutral, and 3 for 
agree). First, the average ranking values of six members in four dimensions were calculated. The results indicated: 
Immersion (M = 2.64, SD = 0.81, FL = 0.79), perceived social interactivity (M = 2.69, SD = 0.80, FL = 0.75), perceived 
humanness (M = 2.60, SD = 0.69, FL = 0.91), and perceived value (M = 2.64, SD = 0.81, FL = 0.89). Those descriptive 
values of constructs showed that the averages were at a high level from the teachers’ perspectives (see Table 1). 


Table 1 
Means, Standard Deviations, Factor Loadings 


Constructs M SD FL 
Immersion 2.64 0.81 19 
Social interactivity 2.50 0.68 he) 
Perceived humanness 2.69 0.80 91 
Perceived value 2.60 0.69 89 


Second, the values of Kendall's W related to four constructs were analysed (see Table 2); the W value of im- 
mersion was .853, ***p < .001; the W value of perceived social interactivity was .836, ***p < .001; the W value of 
perceived humanness was .495, **p < .01; and the W value of perceived value was 994, ***p < .001. All teachers did 
reacha high level of agreement on the four constructs, indicating that they positively recognized the quality of live 
streaming for students to participate in learning science using the inquiry approach. However, the total W value 
was .705, ***p < .001, indicating that the coefficient of concordance was confirmed, and the W value of perceived 
value for students to join this learning approach was the highest. That is, the quality of live streaming for inquiry- 
based science learning can be highly accepted. 
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Kendall's W Coefficient of Concordance Analysis 


Constructs N Kendall’s W s df p 
Immersion 6 853 46.079 9 < .001 
Perceived social interactivity 6 836 45.170 9 < .001 
Perceived humanness 6 495 26.746 9 <.01 
Perceived value 6 .994 53.660 9 < .001 


When teachers reported on students’ immersion, social interactivity, humanness, and learning value, they 
noted that the four constructs were relatively stable for rural students. The results indicated that the teachers 
reported a level of 2.69 on the 3-point Likert scale for students’ humanness, which was the highest rating among 
the four factors. Regarding immersion, teachers reported that students having access to situation-specific engage- 
ment, the students’ immersion level was 2.64 on a 3-point Likert scale and was the second highest rating among 
four variables. Briefly, teachers’ perception of using live-streaming for rural students can be conceptualized using 
a four-dimensional conceptualization, where the P-D/O-Q/D-E-T model may be applied. 


Discussion 


Measuring performance is important to gain a better understanding of a system's capabilities, according to 
Gadamer (2008). Adopting the Gadamerian perspective, in this study, the importance of interpretation was care- 
fully considered. This involves examining and thinking about things from various viewpoints to determine the 
significance of the meaning of the learning that the participants offered (Du Plessis & McDonagh, 2021). 

The level of experienced immersion depends on the representation’s accuracy and the level of social inter- 
activity. It is not an inherent trait (Petersen et al., 2022) but implies that immersion should be noted in learning 
concepts (e.g., Yeo et al., 2020). Meanwhile, IBSLA delineates context and meaning through the live-streaming 
environment that facilitates students’ immersion in learning science to successfully complete the live-streaming 
learning activity. The result of this study revealed that, from the observation of their teachers, students can im- 
merse themselves in IBSLA. 

In the live-streaming case of IBSLA, participants in a social framework could navigate presentation files through 
the guidance of the streamer. Moreover, the streamer proposed questions for students to predict and discuss with 
other participants to provide social interactivity in the live-streaming environment. Learning goals may be hindered 
in atechnology-driven learning environment (Petersen et al., 2022). Participants perceived that social interactivity 
is an important component affecting their engagement in the live-streaming learning process. The result of this 
study showed that the social interactivity perceived by teachers was at a high level. 

Collaboration and discussion are important aspects of online live streaming, which brings people together to 
complete tasks. To facilitate IBSLA in live streaming, accessible tools and resources should be made available. Tulk 
Jesso et al. (2020) studied participants’ beliefs about the interactive agent and the extent to which they perceived 
the interaction with an agent as a ‘real person: They found overall perception and the cues in an environment 
that participants used to determine humanness. In this study, the researchers provided participants with various 
tools and materials, including iPads and experimental objects, to encourage active learning through live stream- 
ing. Engaging students in activities that utilize shared tools, such as Zoom, can help them perceive humanness 
in an interactive model of IBSLA. Moreover, effective visuals and clear voice communication are crucial for active 
engagement in learning activities, which is vital for developing students’ perceptions of humanness. Adopting a 
new ontological model, Isrow (2022) suggested that explaining humanness in specific activities is better. Taking 
IBSLA as a specific learning approach, this study showed that teachers perceived high humanness in IBSLA live- 
streaming, which can activate IBSLA as a shared cognitive base for learning activities. 

Rural primary schools in many countries have poor educational resources and few qualified science teachers. 
For example, in Australia, as in many countries, rural students lag behind urban students in science achievement 
and interest (Echazarra & Radinger, 2019). There have been some attempts to introduce information technology 
to many research projects to solve some of these problems. In reviewing related research from an international 
perspective, an interesting finding emerges, indicating that some studies support the notion of a ‘deficit model’ 
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for teaching rural school students (Abrams & Middleton, 2017). Some research has been analysed to determine if it 
adopted appropriate approaches to introducing e-learning to rural students (Lindfors & Pettersson, 2021); however, 
few studies have used live-streaming to implement an inquiry-based science learning model. According to Singh 
et al. (2021), users’ intention to continue using streaming services is primarily influenced by their perceived value 
and enjoyment. Clarke (2012) suggested that individuals’ responses to live streaming can be understood through 
cognitive appraisal, which can also affect their performance. As a result, this study explored how teachers perceived 
the value of using live streaming to conduct IBSLA and found that they considered it to be highly valuable. 


Conclusions and Implications 


Knowing the psychological factors that affect the implementation of live-streaming for science inquiry learn- 
ing can not only aid in comprehending the characteristics of mutual understanding in human conversations but 
can also assist in developing and enhancing live-streaming technology to make it more human-like in the learning 
process. In this study, the performance of using live streaming for implementing IBSLA was evaluated based on 
four constructs: immersion, social interactivity, perceived humanness, and value from the teachers’ perspective. The 
study's results provided advanced insights into adapting live-streaming for inquiry learning and suggested that it 
can facilitate rural students’ effective science learning. While this paper by no means endeavours to provide answers 
to the use of live streaming for rural primary school science learning, this study touched on the tensions of lacking 
resources and qualified science teachers as it explored how non-science teachers experienced live-streaming learn- 
ing by linking their perceptions to the inquiry-based model and provided the knowledge when using this model. 

Learning with live streaming, in this study, refers to students engaging in learning from another place other 
than school. However, science inquiry learning with live-streaming learning refers to the type of teaching that occurs 
when students are co-located in the rural school’s physical classroom, and the teacher offers remote instruction 
from another location. According to the research, momentary immersion reflects situation-specific engagement. 
However, current conceptualizations seem to be at each phase of P-D/O-Q/D-E-T, which may result in students only 
partially reflecting on the understanding of the digital context or on what the live-streaming teachers say. Thus, 
those factors may lead to an understanding of the growing usage of live streaming for science teaching. Moreover, 
to equalize the learning materials in science learning units, the researchers sent the experimental material before 
the live-streaming class. In this case, it is suggested that only those science units with accessible materials can be 
easily adapted according to the P-D/O-Q/D-E-T model. Additionally, teachers found that the P-D/O-Q/D-E-T model 
can be successfully applied through live-streaming in short science courses, such as those covering surface tension 
or gravity. For teachers who require live-stream science teaching, the focus is on identifying which science learning 
content would be appropriate to be designed with the P-D/O-Q/D-E-T model. 

Observational research is abundant and influences education practice. However, it frequently generates 
unreliable findings (Wang et al., 2015). Inherent limitations of this method include generating bias and confound- 
ing means so that causal inferences cannot be reliably drawn. Because of the limitations, future evidence may be 
collected during experimental studies to identify how participants’ cognition and affective process influence their 
learning effectiveness. Another limitation occurred with the Internet system, Google Meet, which was used as the 
platform in this study. Google Meet can accommodate many participants from many locations, but if there are 
too many participants, the screen of each participating group would become too small for students to clearly see 
other participants’ reactions. It would, therefore, be helpful to study how many participating groups from differ- 
ent locations are most suitable to conduct live streaming in short science courses with the P-D/O-Q/D-E-T model. 
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Abstract. STEM education, which includes 
science, technology, engineering and 
mathematics, has been expanding for 

the past two decades. This study aimed 

to map new trends and the possibility 

of implementing digital game-based 
learning (DGBL) in STEM education. For 
this purpose, a systematic literature review 
was conducted. The resulting sample was 
further selected according to PRISMA 
guidelines, with screening and eligibility 
processes conducted based on the 
inclusion criteria defined concerning the 
research objective. This review consisted 

of twenty-eight studies. The findings 
revealed a growing interest in DGBL 

in STEM education from 2018 to 2023. 
Furthermore, most studies have focused on 
the K-12 education system and universities. 
According to the review, educational 
games for digital learning and simulation 
technology are the most promising tools 
used in research. The analysis is launched 
by studying the effects that influence 

the increase of student motivation in 

DGBL teaching STEM education. The 
findings support the conclusion that prior 
experience in gaming has a positive impact 
on increasing students’ motivation to 
learn in DGBL STEM teaching. In addition, 
students’ previous knowledge of a STEM 
subject increases engagement and 
motivation. Implementing educational 
computer games, therefore, showed a great 
interest in students in STEM education. 
Keywords: digital game-based learning, 
students’ motivation, STEM education, 
systematic literature review 
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Introduction 


STEM education is increasingly popular and present at all levels of 
education. The acronym STEM stands for four scientific disciplines, science, 
technology, engineering, and mathematics (Aykan & Yildirim, 2022). These 
scientific disciplines can be observed and implemented independently, the 
most common case in many educational systems, or as an interdisciplinary 
combination of individual STEM disciplines (Nadolny et al., 2022). Learning 
STEM disciplines presents various difficulties to students, primarily due to 
scientific disciplines’ complex, multidimensional and abstract nature. The 
solution to these difficulties is introducing digital video games into the 
curriculum. It has been noted that digital games can successfully meet the 
challenges of STEM education. Researchers have shown that integrating 
digital games into STEM education serves various purposes (Fadda et al., 
2021; LeRoy 2022; Hussein et al., 2022). 

Researchers have noticed that introducing digital games into STEM 
education achieves multiple intents. Along with promoting and developing 
STEM education, the critical role of motivation as a driving force and outcome 
of STEM learning is widely recognized. Dedication and a high level of student 
motivation are necessary for successful learning. Digital game-based teaching 
has quickly found its place in many educational systems (Gui et al., 2023; Hus- 
sein et al., 2022; Ross et al., 2022). The increasing prevalence and availability 
of video games among children and young people have led to the need to 
see the possibility of using digital video games beyond entertainment and 
leisure. As a recent trend in education, it has shown positive effects on student 
engagement and motivation. In recent years, digital educational games have 
become more common. There are specially designed digital educational 
games. Conversely, various digital or video games intended for commercial 
purposes have undergone remastering to cater to the educational sec- 
tor. Minecraft, for example, has been adapted into an educational version 
known as Minecraft Education Edition. Educational video games, especially 
becoming popular in the last fifteen years, are the subject of interest to many 
researchers, although the term has yet to be recent (LeRoy 2022; Tablatin et 
al.,2023; Vostinar et al., 2022). 
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Two independent studies by Landers and Welbers showed positive results of video game-based learning 
compared to traditional learning, where students receive all the necessary instructions from the teacher (Landers, 
2014; Welbers et al., 2019). So far, several studies have been published on the effects of STEM education on stu- 
dents learning motivation. However, there is some disagreement among researchers. Some talk about the positive 
impact of digital video games in STEM education, such as Hussein et al. (2022) have pointed out that the use of 
digital games has a positive effect on increasing students’ motivation to learn mathematics. On the other hand, 
this claim is supported by other authors such as Ross et al. (2022), Gladstone and Cimpian (2021) and Fadda et al. 
(2022). However, several researchers reported that the introduction of video games into the teaching process of 
STEM subjects did not affect the increase in student motivation. Dondio et al. (2023) have pointed out that playing 
games in mathematics lessons did not significantly improve students’ motivation to learn mathematics. From this 
discrepancy arises the need for a more extensive assessment of whether digital-game-based learning positively 
impacts students’ motivation in STEM subjects. This research aims to analyze scientific studies on the effect of digital 
game-based learning in STEM education on student motivation. Given that the STEM concept is quite complex, this 
study considered sub-disciplines, mathematics, science, and engineering (especially programming and computer 
science). As educational games are widely used, there is a clear need to analyze how the effect of digital game-based 
learning in increasing motivation has been studied and how such studies can inform and encourage teachers and 
practitioners to improve it. For theorists, the degree of acceptance of digital video games for educational purposes 
is of great importance, on the one hand, but also the response of teachers and students, which directly influence 
the effective further development and use of educational video games in teaching practice. 


Aim 


This analysis has aimed to address the problems of incomplete and imprecise empirical studies on student 
motivation in digital game-based learning in STEM education. Considering that STEM education is a composition 
of four scientific disciplines, it should be taken into consideration what is the attitude of the students towards 
individual disciplines, whether they are more biased towards a particular subject, whether attention was equally 
maintained in the classes of mathematics, science, technology and/or engineering. In order to maintain objectivity 
during the analysis, the following research questions guided the conduct of this study: 

1. Does engagement and motivation increase when utilizing digital game-based learning in STEM edu- 
cation? 

2. Does the level of student's knowledge of a given STEM discipline increase or decrease learning moti- 
vation? 

3. Does prior knowledge about the game affect students’ motivation to learn? 


Theoretical Backgrounds of Digital Game-Based Learning in STEM Education 


One of the essential features of digital educational games (DEGs) is their potential for application outside of 
STEM education and their benefits in improving student motivation for learning and engagement during learning. 
Many authors define educational digital games and digital game-based learning differently (Clark & Mayer, 2016; 
Prensky, 2003; Sailer & Homner, 2020). One thing is expected in the sea of different interpretations of explanations 
and definitions. Essentially, educational digital games are designed to encourage and improve the student's achieve- 
ment as an individual and then the student as an equal member of the community class. Regarding STEM educa- 
tion, digital games are significant for student achievement and cognitive development. Educational digital games 
provide an innovative, exciting and interactive approach to learning, creating an environment that encourages 
student activity while allowing students to develop problem-solving skills and critical thinking, which is essential 
for achieving STEM education goals. In this way, the increasingly significant development of 21st-century skills 
and the expansion of STEM learning and knowledge are encouraged (Stehle & Peters-Burton, 2019). At the center 
of game-based learning is a game (digital, computer, educational game) that supports the learning of students 
with different competencies and prior knowledge. Educational games or video games modified for educational 
purposes with their interface positively affect the sustainability of students’ attention. An important feature is that 
they are often flexible, and teachers can respond relatively quickly to teaching challenges. Feedback is essential for 
the learning process and is available at any time. Numerous scientists suggest that using educational games could 
be of multiple benefits for improving students’ knowledge levels and the overall outcome of teaching (Gui et al., 
2023; Vanbecelaere et, al., 2019). By introducing educational games into the teaching process, student engagement, 
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learning achievements and student motivation for learning increase (Arztmann et al., 2022). 

Learning is a multifaceted cognitive process, and numerous scholars emphasize the significance of motivation 
as a critical factor in successful learning (Fortus & Touitou, 2021). While it is widely acknowledged that motivation 
is indispensable for quality education, educational systems worldwide have grappled with student apathy. An 
essential aspect of learning based on digital educational games is the motivation of students to learn. Motivated 
students tended to engage during the teaching process (in the classroom or online). 

The growing number of students engrossed in video games has prompted researchers to explore the poten- 
tial of games as an innovative educational tool. Games offer the opportunity to infuse learning with enjoyment. 
Reward systems, encompassing scores, assets, permissions, reputation, and more often drive motivation within 
games. These rewards typically stimulate and reinforce extrinsic motivation, derived from desired outcomes or 
external pressures, rather than intrinsic motivation, which stems from an individual’s inherent interest in the activ- 
ity. However, it is worth noting that extrinsic reward systems are associated with superficial learning rather than 
deep processing. Understanding student motivation during learning based on educational digital games in STEM 
education is challenging due to the nature and multidisciplinarity of the STEM concept. In the last two decades, 
many researchers have studied student motivation during the learning process in STEM education (Gustiani, 2020; 
Chiang et al., 2022; Ross et al., 2022). Earlier studies showed that students’ motivation decreased with the transition 
to a higher grade, so students in higher grades avoided STEM subjects, and dropouts were mainly observed after 
the completion of K-12 schooling (Kuo, 2007; McDonald, 2016; Osborne et al., 2008; Vogel et al., 2006). Educational 
computer games have been recognized as a good way of providing a more engaging learning environment for 
acquiring knowledge. Learning supported by digital educational games has multiple effects on student achieve- 
ment. Many researchers have reported that the existing potential of the new mediums can be used as a tool in 
education to improve students’ motivation and provide a better learning experience (Puspitarini & Hanif, 2019). 
In general, students who possess a stronger sense of intrinsic motivation often gravitate towards digital learning 
environments. This preference arises from the ability to work at their own pace without relying on teacher support 
or the classroom environment for motivation (Taylor & McNair, 2018). Wang et al. (2022) confirmed that students 
at different educational levels react differently to the game but that the level of motivation did not depend on the 
type of game but on the difficulty of the teaching material. Motivation is closely related to achievement. Wang 
(2022b) found that students performed better and were more motivated to learn mathematics when the teaching 
material was presented using a digital educational game. Even students with worse prior knowledge showed more 
than expected interest in getting involved in the teaching process. STEM is based on a cohesive learning paradigm 
based on real-world applications in an interdisciplinary field. The concept of STEM disciplines is in a synergistic 
relationship, and it is often the case that if a student is interested in mathematics, it does not necessarily mean that 
he will be equally motivated to learn science. Here comes a slight departure from the assumption that STEM is an 
ideally conceived interdisciplinary paradigm (Saricam & Yildirim, 2021). Although educational games are generally 
believed to enhance student motivation, the impact of these games on motivation has yielded inconsistent and, 
at times, contradictory findings. 

Moreover, more research needs to explore the influence of intrinsic and extrinsic motivations on game-based 
learning. Additional design measures, such as collaboration, have been implemented to improve peer communi- 
cation and enhance the effectiveness of educational games. However, they have yet to demonstrate significant 
changes in motivation levels (Van der Meij et al., 2020). Situational games have shown promise in boosting students’ 
motivation and performance, while competitive games have been found to frustrate students (Yang et al., 2020) 
potentially. Adaptive computer games, which assess student performance and adjust difficulty levels, accordingly, 
have not significantly improved children’s motivation (Vanbecelaere et al., 2019). Although various studies have 
shown the benefits of using educational digital games in enhancing students’ motivation to acquire STEM knowl- 
edge, a few have denied any educational digital games’ contribution to STEM education (Gustiani, 2020; Safarud- 
din et al., 2020). In addition to motivation, students’ activity should also be considered, which is directed towards 
learning, i.e. acquiring, and upgrading STEM knowledge. Thus, engagement, motivation and learning represent a 
unity directed towards learning based on digital educational games (Breien & Wasson, 2021). 


Indicators of Student Motivation STEM Education 
As a multidisciplinary concept, STEM must be understood as something other than material that is only con- 


veyed through verbal and face-to-face teaching methods. All set goals can be achieved only when the student 
is active and equal in the teaching process. This is of particular importance when it comes to STEM subjects. Mo- 
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tivation during the process of acquiring knowledge is fundamental for the learning outcome. Different views on 
motivation are often emphasized in the literature. According to Herder and Rau (2020), motivation is defined as 
a set of processes aimed at maintaining and encouraging students’ activities directed towards the ultimate goal. 
Motivation is a complex and ongoing research subject, especially in disciplines such as tree education. We live in a 
world of constant technological progress; current information will already be outdated tomorrow, so we are trying 
to find new methods to transmit knowledge and information. 

Motivation plays a leading role in the learning and teaching process, so innovative approaches to learning 
are needed to increase motivation. This is important because the factors that influence motivation are constantly 
changing and becoming outdated. It is necessary to maintain continuity to maintain the quality of education (Tisza 
et al., 2021). There are two types of motivation (Filgona et al., 2020). Internal motivation, as one type of motivation, 
is a characteristic of individuals. External factors do not determine internal motivation but directly depend on the 
needs and aspirations of the individual. In contrast to intrinsic motivation, extrinsic motivation is the desire to 
engage in an activity to achieve positive outcomes, such as rewards (often in the form of positive evaluations) or 
punishments (often in the form of negative evaluations). 

The creators of video games know very well the psychology of the shell of the personality; this has proven to 
be especially important when it comes to educational digital games. Educational digital games have built-in moti- 
vational attributes that attract and maintain the player’s or student's attention. Intrinsic motivation is emphasized 
for the use of digital gamers for educational purposes. With a well-designed story and game environment, some 
digital games gained worldwide popularity today. One example is Minecraft, which indicates that Minecraft has 
developed its educational version, Minecraft EDU. 

Research has shown that intrinsically motivated students are more likely to persist in the face of learning 
challenges than extrinsically motivated students (Byun & Joung, 2018; Breien & Wasson, 2021; Liao et al., 2019). 
The central focus is the relation of the motivational appeal of digital game-based learning in the context of STEM 
education. In the domain of curriculum design, DGBL can serve as a powerful motivational tool for students and 
teachers, as it encourages peer collaboration and support in teaching. 


Research Methodology 


The preparation of this systematic review followed the Preferred Reporting Items for Systematic Review and 
Meta-Analyses (PRISMA) guidelines by Moher et al. (2015), as shown in Figure 1. This systematic review process 
included the following stages: identification, screening, eligibility, and inclusion. 


Identification 


The Science Direct, Springer Open and Google Scholar databases were used in the identification process to 
search for relevant articles. For the search, keywords related to digital game-based learning, student motivation, 
gamification, digital educational games, and STEM education were used. A combination of keywords and Boolean 
operators AND and OR was used. After the initial process, the result was 201 articles from Science Direct, 153 from 
the Springer Open database and 127 from Google Scholar. However, 36 duplicate studies were removed, leaving 
446 articles found. To guarantee that only studies meeting the standards of the scientific community were identi- 
fied, we focused exclusively on peer-reviewed articles and excluded grey literature. 


Screening 


Considering the different databases and the relevance and eligibility of the published articles, the abstract of 
each article was first reviewed individually. The second stage continued with the screening process based ona set 
of criteria shown in Table 1. This study included the publication timeline between the years 2018 until the year 2023. 

Only article journals and dissertation theses that used the English language were included, focusing on issues 
in STEM education and digital game-based learning. For the first screening process, out of 446 articles, 232 articles 
were excluded. Research topics were STEM education, including science, physics, mathematics, technology, and 
engineering education. The papers included in this research encompass empirical research, studies with control 
and comparison groups, as well as meta-analyses. 
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Table 1 
Screening Process 
Criteria Inclusion Exclusion 
Publication timeline 2018-2023 2019 and before 
Document type Article Conference proceedings, chapters in books, 


dissertations, books, etc. 


Language English Non- English 
Nature of the study Concentrate on the issues of learning Not focus issue of learning supported by digital 
supported by digital games. games. 


Eligibility and Inclusion in Research 


Since the end of the first screening cycle, 148 papers have been excluded, and 166 papers have been completed. 
The next step is the second level of screening, the paper quality check. The criteria based on which the screen- 
ing was performed are shown in Table 2. It is necessary to see whether the goals and objectives of the research 
are clearly defined and then implemented. The following substantial question is how the study was conducted. 
During the analysis of each article, the methodology used must be considered, including the method of analysis, 
collection, presentation of data, qualitative and quantitative, and how the research data were communicated and 
presented. It is also necessary to look at the target group covered by the research. 


Table 2 
The Criteria Quality Assessment of Articles 


No Criteria Descriptions 
1 Study context Did the article effectively communicate the study's planning, implementation, and develop- 
ment aspects? 
2 Objectives and purposes Were the aims and objectives of the study clearly articulated in the article? 
3 Methodology Does the article comprehensively explain the research methodology, including the framework, 


data collection, and data analysis? 


4 Data Are there supporting materials such as schedules, interview or focus group transcripts, and 
observational feedback? Is the information presented in a clear and analyzed manner? 


5 The results of the study are Were the results of the study validated? Did the researcher utilize peer review, feedback, or 
validated other mechanisms to confirm the analysis? 
6 Sample Does the article offer an adequate sample size for the research conducted? 


The next screening involves evaluating the quality of the articles. In order to make an authoritative assessment 
of the quality and suitability of architects, a table examining seven criteria was used. Each criterion is additionally 
evaluated from 1 to 4, depending on the degree of satisfaction of the criteria. If the score is 1, it means that it does 
not meet the standards, a score of two means that the standards have almost been achieved, a score of three 
means that the criteria have been met, and a score of four means that the required criteria have been exceeded. 

After further analysis and evaluation of each article, each could receive a minimum number of points of 7 (no 
criteria met 7x1) or a maximum of 28 (all criteria were evaluated with 4 points each). Screening excluded articles 
with less than 14 points, so they only met some criteria. 
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Table 3 
Quality Assessment Table 


Objectives and 
purposes 


Theoretical or 


conceptual frameworks 


Participants 


Review of literature 


Methods 


Results and conclusions 


The problem, objective, 
rationale, and research 
questions are clearly 
articulated. 


Clearly articulated and 
described in detail. The 
frameworks align with the 
purposes of the study. 


It provides a detailed, 
contextual description of 
the population, sample, and 
sampling procedures. 


Critically examines the state 
of the field, clearly situating 
the topic within the broader 
field. Makes compelling 
connections to past work, 
discusses and resolves 
ambiguities in definitions, and 
synthesizes and evaluates 
ideas while offering new 
perspectives. 


The article provides a detailed 
description of the instruments 
used and their administration, 
along with evidence of 
validity, and reliability. It 

also documents the use of 
best research practices and 
considers potential bias. 


The article provides detailed 
results, utilizes exceptional 
data displays, and the 
discussion connects the 
findings to past work. It also 
proposes future directions for 
research. The conclusions 
effectively address the 
problem or questions at hand. 


The problem, objective, 
rationale, and research 
questions are adequately 
articulated. 


Articulated and aligned with 
the study’s purposes. 


The article provides a detailed 
description of the population, 
sample, and procedures in a 
contextual manner. 


Discusses what has and 
has not been done, situates 
the topic within the broader 
field, connects to past work, 
defines critical vocabulary, 
and synthesizes and 
evaluates ideas. 


The article describes the 
instruments used and their 
administration, provides 
evidence of validity or 
reliability, and shows some 
evidence of employing best 
research practices. Potential 
bias is considered. 


The article presents complete 
results, utilizes good data 
displays, and the discussion 
connects the findings to 

past work. The conclusions 
adequately address the 
problems or questions. 


The problem, 
objective, rationale, 
and research 
questions are poorly 
articulated. 


Implied or described in 
vague terms or fails to 

align with the purposes 
of the study. 


The article provides a 
basic description of the 
sample and procedures. 


Minimally discusses what 


has and has yet to be 
done, vaguely discusses 
the broader field, makes 
few connections to 

past work, needs more 
synthesis across the 
literature, and provides 
a minimal evaluation of 
ideas. 


The instruments are 
described, but more 
evidence of validity 

or reliability must be 
provided. The use of 
questionable research 
practices may be 
present. 


The results provided 
are essential, and data 
displays must be used 
more. The discussion 
needs to establish 
connections to past 
work. The conclusions 
merely summarize the 
findings. 


The problem, objective, 
rationale, and research 
questions are 
incomplete. 


Absent (no mention or 
description of theoretical or 
conceptual frameworks). 


The description of the 
sample and procedures 
needs to be completed. 


It fails to discuss what has 
and has yet to be done, the 
topic is not situated within 
the broader literature, and 
there are no connections 
made to past work. 


The description of 
instruments and their 
administration needs to be 
completed. 


The results and 
conclusions must be 
completed, needing more 
detail or information. 


The screening and selection process was divided into three main steps according to the PRISMA flow dia- 


gram (Page et al., 2021). In the first step, only the titles and abstracts of the articles were considered. Studies were 
included if their abstract was written in English and contained terms such as “learning based on digital games’, 
“digital games and STEM education’, “student motivation in DGBL teaching’, and reported on the results of empiri- 
cal research. Studies using methodologies other than surveys, not published in peer-reviewed academic journals, 
from areas other than education research, or not considering any K-12 level were excluded. After the first step, 


144 documents were excluded. 
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Figure 1 
PRISMA Flow Diagram 
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The following screening step included checking the availability of documents. Excluded were those studies 
that could not be found entirely available online, or even after contacting the author, we did not have access to 


the article (= 42). 


Finally, the last step involved reviewing the available articles and applying our exclusion and inclusion criteria. 
Based on the already mentioned exclusion criteria, the final number of excluded articles is = 144. Only studies that 
empirically addressed the issue of integrating learning based on digital games and examining students’ attitudes 
and levels of motivation for learning based on digital games were included. During this phase, 28 studies were 
included. The following chart presents published articles by year (see Figure 2). 


Figure 2 
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Data Extraction and Content Analysis 


After a systematic review of the literature and the identification of 28 relevant studies from 2018 to 2023 
(published by May 2023), an analysis of them was carried out using content analysis, with the focus being on the 
content and way of processing research questions (see Table 4). 


Table 4 
Summary of Included Articles 


Year Researcher Findings 

2023 Guietal. STEM digital games are more effective for developing cognitive skills than facilitating knowledge acquisition. 

2023 ~— Al-Said, K. It was concluded that students in interventions with games achieve significantly better learning results and 
report greater motivation and change in behavior compared to students in traditional classrooms. 

2023  Tablatin et al. Persistence in problem-solving relies on students’ proficiency and familiarity with the game environment. 

2023 Yang No single educational game can be used for all STEM disciplines and all levels of education. 

2022 Wangetal. Increased level of students’ cognitive abilities during learning based on digital games. 

2022 Nkadimeng & Ankiewicz Visualization reduces the abstractness of concepts and increases student motivation. 

Knowledge of educational games is necessary to increase the possibility of success and increase students’ 
level of achievement in developing knowledge and skills in digital-game-based learning. 

2022 = ‘Tayetal. Learning with digital technology-facilitated empathy: an augmented reality approach to enhancing students’ 
flow experience, motivation, and achievement in a biology program. 

2022 Wu, Liu, & Huang The research discovered that, in STEM learning, one should maximally utilize a student's attitude, intrinsic 
motivations, and cognitive load to obtain knowledge in STEM fields effectively. 

2022 = Arztmann et al. Reported that there was no difference in the impact of play-based learning between boys and girls but that 
they showed equal interest and motivation in DGBL. 

2022  Pogan etal. It increases students’ motivation to learn and understand abstract concepts. 

2022 Wangetal. Digital game-based learning shows better results than other teaching methods. 

2022 McLaren etal. How instructional context can impact learning with educational technology: Lessons from a study with a digital 
learning game. 

2022 LeRoy The student's active involvement in the learning process is crucial, as it fosters a sense of ownership and 
belonging. This, in turn, positively impacts motivation, personal connection to the material, and a deeper 
comprehension of the information and the methods employed to acquire it. 

2022 Andersen & Rustad Digital educational games increase students’ activity levels and desire to learn. 

2022 Mawas etal. The game's influence on student motivation may vary between the two countries due to the potential influence 
of students’ pre-existing attitudes before the courses. 

2022 Zhang etal. Prior knowledge of students positively influenced the motivation for digital game-based learning in stem 
disciplines. 

2021 Sdmer, Moreira, & Casado The Kahoot application, with elements of a digital educational game, showed high average effectiveness 
in evaluating motivation indicators, actively involving students in the learning process, improving content 
processing and increasing experience, motivation, attention and satisfaction. 

2021 + Faddaetal., The findings indicate that digital games are more practical than conventional teaching practices. 

2020 = Sabirli & Coklar The effect of educational digital games on education, motivation, and attitudes of elementary school students 
against course access 

2020 Bertram, L. Game-based learning interventions should offer students active opportunities for self-directed learning and 
positively influence attitudes, emotions, motivation and engagement. 

2020 Wang & Tahir The transition from traditional to online teaching led to decreased student motivation. With the introduction of 
DGBL in online teaching, student motivation has strengthened. 

2020 _ Priyaadharshini et al. Game-based learning helps all levels of learners, from beginners to experts. 

2019  Ortiz-Rojas et al. Prior experience playing games encourages students towards digital game-based learning in stem education. 
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Year Researcher Findings 

2019 Hosseini et al. Game-based game learning strategy improves the self-efficacy of the learners. DGBL enabled a clearer insight 
into student engagement and, thus, motivation. 

2018 Byun & Joung Learning based on educational digital games has an encouraging effect on improving students’ STEM skills. 

2018  Touati & Baek Previous experience playing commercial games increases interest in learning based on educational games. 

2018  Zhaoetal. With traditional approaches to education, many students consider science subjects problematic and are often 
discouraged during the learning process. 

2018 Prasetyo & Napitupulu Digital game-based learning did not improve learning achievement and motivation in mathematics classes in 


K-12 students compared to traditional methods. 


Table 5 provides an overview of all analyzed empirical research with critical data on the works. The table shows 
the age of the respondents, their number, and the country where the research was conducted. The following is 
data on the critical research questions and their answers (positive, negative, impact cannot be determined). It is 
also indicated if the paper should have discussed one of the questions. 


Table 5 
Summary of Included Articles 


Years Researcher 
2023 Gui et al. 

2023 Al-Said, K. 

2023 Tablatin et al. 

2023 Yang 

2022 Wang et al. 

2022 Nkadimeng & Ankiewicz 
2022 Tay et al. 

2022 Wu, Liu, & Huang 
2022 Arztmann et al. 

2022 Pogan et al. 

2022 Wang et al. 

2022 McLaren et al. 

2022 LeRoy 

2022 Andersen & Rustad 
2022 Mawas et al. 

2022 Zhang et al. 

2021 Somer, Moreira, & Casado 
2021 Fadda et al., 

2020 Sabirli & Coklar 

2020 Bertram, L. 

2020 Wang & Tahir 

2020 Priyaadharshini et al. 
2019 Ortiz-Rojas et al. 

2019 Hosseini et al. 

2018 Byun & Joung 

2018 Touati & Baek 

2018 Zhao et al. 

2018 Prasetyo & Napitupulu 
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Research Results 


Articles included in the sample were published from 2018 to May 2023. However, the random sample included 
only six articles published between 2017 and 2018. To respond to the research questions, the results section consists 
of three topics related to the level of education, country or province of research and type of educational digital games. 

RQ1: Is the level of motivation higher when performing digital game-based learning in STEM education 
compared to teaching dominated by teaching aids such as textbooks, teaching sheets, demonstration methods, 
and teachers’ oral presentations? 

This systematic analysis includes 28 relevant articles on STEM education and student motivation. A large num- 
ber of studies (= 21, 75%) confirm that digital games in STEM education increase the level of student motivation. 
A few studies (= 7, 24%) denied digital games’ contribution to increasing students’ motivation. The contribution of 
DGBL varied by level of education. Studies have shown that the level of motivation is higher among K-12 students 
(= 19, 67.8%) of the educational system than among university-level students (= 9, 32.2%). The level of motivation 
also depended on the lesson’s content covered in class. Students were more motivated for more straightforward 
concepts (= 10, 35.7%) and lessons than abstract and unfamiliar concepts (= 7, 25%) (see Figure 3). In favor of DGBL, 
there are also data that from the total number of articles included in this analysis, as many as 54% of students over 
15 years and 74% of students under 15 years want to use technology more, and therefore DGBL in the classroom 
when learning STEM subjects (see Figure 4). 


Figure 3 Figure 4 
Educational Level and Increasing Motivation Selection for DGBL Based on the Age of the Student 
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The 28 articles that met the inclusion criteria had at least two STEM disciplines ( = 17, 60.7%), while several 
dealt only with programming as a STEM discipline ( = 6, 21.5%). A significant contribution was made by analyzing 
articles that looked at the overall contribution of the STEM concept education in DGBL, not referring to any specific 
STEM discipline (= 5, 17.8%). No articles were found that focused on all STEM disciplines. 


Figure 5 
Articles Based on the STEM Discipline in Reviews Studies 
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Several studies deal with the contribution and implementation of technologies suitable for DGBL in STEM 
disciplines (see Figure 6). The author identified two used game technologies directly related to motivation while 
learning STEM subjects from the reviewed literature. Three articles (10.7%) use the Kahoot application, whose 
contribution at the global level is 85% in favor of stimulating and maintaining student motivation. Other findings 
indicate that Minecraft EDU has positively affected student motivation (= 5, 17.8%). 


Figure 6 
Educational Digital Games in STEM Education 
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Considering the above findings, it is possible to confirm that the introduction of learning based on digital 
games contributes positively to increased student motivation than learning STEM disciplines using teaching aids 
such as a textbook, worksheets, and a projector followed by a teacher’s oral explanation. 

RQ2: Does the pre-knowledge of students given STEM discipline increase or decrease motivation? 

One of the critical questions is whether the student's level of knowledge affects motivation. Research has 
shown that an increase in motivation was observed in the majority of respondents who already had prior knowl- 
edge of a specific STEM teaching subject ( = 11, 39.3%). A decrease in motivation was observed in students with 
weaker knowledge or needing adequate digital devices to monitor DGBL (=4, 14.3%). Several articles emphasized 
that the students’ pre-knowledge was not essential for increasing students’ motivation ( = 2, 7.1%) (see Figure 7). 


Figure 7 
The Effect of Prior Knowledge on Student Motivation 
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Introducing educational games into the teaching process led to an inevitable increase in motivation. Stu- 
dents’ prior knowledge of a specific STEM discipline favored the increase in student motivation in teaching STEM 
disciplines supported by Minecraft EDU ( = 3, 10.7%). Students’ prior knowledge combined with Kahoot proved 
good for delivering STEM lessons. A significant increase in motivation was observed among students (= 2, 7.14%), 
while a small number said that prior knowledge did not significantly affect the growth in motivation (= 1, 3.6%). 
Figure 8 illustrates the educational digital games used in the reviewed studies. 
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Figure 8 
The Impact of Earlier Game Playing on Students’ Motivation 
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Twenty-eight articles were included in this analysis from the total sample, 41% of students older than 15 em- 
phasized that prior knowledge of STEM subjects increased their motivation. Meanwhile, 63% of students younger 
than 15 were more motivated for DGBL due to their quality knowledge of STEM subjects (see Figure 9). 


Figure 9 
The Influence of Prior Knowledge on Student Motivation, Age Selection 
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That prior knowledge influenced the motivation for further learning in DGBL STEM learning is confirmed by 
eight (28.6%) articles that included at least two STEM disciplines; six articles (21.4%) emphasize that prior knowledge 
is not crucial for student motivation. It has been shown that prior computer science or mathematics knowledge is 
significant for motivation and further progress in DGBL learning STEM subjects. It was not found that prior knowl- 
edge had equally positive effects on increasing motivation in all STEM disciplines. 

RQ3: Does prior knowledge about the game affect students’ motivation to learn? 

Previous experience and knowledge of playing a game can be essential for further progress during learning. 
It turned out that students who had previous experience playing games found it easier and faster to get used 
to DGBL in STEM lessons than those who had not played games before ( = 6, 21.5%). Students with no previous 
experience playing games did not show motivation for further progress and engagement in DGBL teaching STEM 
subjects ( = 2, 7.14%). However, some have shown that previous experience of playing games is not decisive for 
increasing or decreasing students’ motivation in DGBL teaching of STEM disciplines, i.e. 3.6% of the reviewed 
literature (see Figure 10). 
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Figure 10 
Effects of Prior Gaming Experience on Motivation 
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Since video games are a prevalent form of entertainment among students, the experience of playing games 
proved to be a good practice for increasing motivation in DGBL. Minecraft EDU (see Figure 11) significantly increased 
student motivation, especially among those with previous experience with this computer game's commercial 
version (= 7, 25%). 


Figure 11 
Impact of Prior Minecraft Gameplay Experience on Student Motivation 
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However, motivation remained the same or decreased in students without prior experience playing games. These 
students often gave up their studies ( = 2, 7.14%). Students older than 15 years showed that their previous experi- 
ence of playing games positively affected their maintenance and increased motivation for learning (= 5, 17.8%). No 
statistically significant contribution of experience was recorded for students under 15 years of age. 


Discussion 


Findings from this study may have implications for improving student motivation in digital-game-based learn- 
ing in STEM Education. 

Student motivation during learning is one of the most powerful driving forces towards achieving goals during 
teaching and learning. The effect of DGBL was investigated in terms of its ability to enhance motivation in learning 
STEM disciplines among students in the K-12 education system. To prevent a decrease in student motivation, educa- 
tional technologies and teaching aids are constantly researched and sought after. Since each student is a microcosm 
for himself, creating and designing a quality environment is necessary to encourage the student to get involved in 
the teaching process. 

The present study results suggest notable discrepancies between students who have played digital games before 
and those who have yet to. It implies that students with experience playing games are more open to adapting to 
new learning environments, leading to better learning outcomes. The findings of the study are in line with Prasetyo 
and Napitupulu (2018), who stated that prior experience in playing digital games has been found to contribute to 
an increase in motivation to learn. Still, it cannot be considered as the sole factor that increases student motivation. 

On the other hand, due to the very nature of STEM disciplines, students are more suited to an innovative ap- 
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proach, the implications of which can be seen in increasing motivation for learning. The minimizing of the traditional 
approach, which includes traditional teaching aids such as blackboards and textbooks, provides space for the integra- 
tion of educational digital games and thereby establishes student interaction. 

This result is similar to the results of Al-Said et al. (2023). For example, Al-Said et al. (2023) found that the inter- 
action of students with digital educational games is an important factor in increasing motivation and raising the 
working atmosphere. 

There is a significant number of educational digital games available in the market today. In this study, Minecraft 
and Kahoot were selected as representative examples. The analysis revealed that the educational version of Minecraft, 
acommercial game, has multiple positive effects on student motivation during the learning process. Many students 
showed a keen interest in learning through this educational digital game. The findings of this study are consistent 
with the results obtained by LeRoy (2022). Knowing how to play the game is not sufficient for sustaining and starting 
it. The level of motivation is largely dependent on the knowledge of a specific STEM discipline. By using Minecraft 
and Kahoot in her classes, the teacher discovered that the level of motivation consistently increases among students 
who have prior knowledge of specific STEM disciplines. However, it was also observed that prior knowledge of STEM 
disciplines does not significantly contribute to increasing motivation in a small number of students. 

This result is partly similar to the results of Somer et al. (2021). For example, integrating educational digital games 
in STEM education showed outstanding results in improving motivation among students with a solid background 
in STEM disciplines. 

University and K-12 students have different biases, cognitive skills, and prior knowledge. Another important 
indicator for students’ motivation to learn in DGBL STEM education is the age of the students. Research has shown 
that K-12 students are significantly more interested in learning through digital educational games than university 
students. On the other hand, we noticed that older students at the K-12 level have a greater affinity for digital games 
than younger students. The findings of this study were partially confirmed by Gui et al. (2023) because they pointed 
out that older students are at a higher cognitive level than younger ones and that games for educational purposes are 
not crucial for motivation. However, they are for further development of cognitive skills. Another group of researchers, 
Tai et al. (2022), pointed out that both groups of students showed increased motivation. However, K-12 new students 
showed a more noticeable increase in motivation and interest in DGBL classes. 

Apart from imparting knowledge to students, a teacher has an equally important role in promoting the devel- 
opment of student motivation. With the help of modern teaching aids, a teacher's job has become much easier. Our 
study demonstrated that motivation can be improved through digital educational games, provided an appropriate 
selection of games. The most promising outcomes were observed in students with prior experience of playing games 
and a good understanding of STEM subjects. 


Conclusions and Recommendations for Future Research 


According to the study, prior experience in STEM subjects has a positive effect on student motivation. In the realm 
of STEM subjects, prior gaming proficiency does not significantly influence one's abilities. However, these disciplines 
necessitate an alternative pedagogical approach that emphasizes a higher degree of student involvement. Digital 
games offer superior prospects for engagement and learning in comparison to conventional teaching tools, such as 
textbooks, worksheets, and verbal lectures. This is because textbooks are limited in their teaching aims and do not 
offer adequate opportunities for student engagement. Consequently, incorporating digital games in the classroom 
can prove to be an efficacious means of enhancing student engagement. 

The study also found that motivation, academic achievement, and engagement are the most extensively exam- 
ined variables, with a subsequent focus on learning, enjoyment, and other cognitive factors. Academic motivation 
was identified as the most researched aspect, and the analyzed studies confirm that DGBL strategies positively affect 
student motivation. Current knowledge and findings indicate that students are more motivated to acquire knowledge 
and skills in DGBL STEM education classes than in classes not customized to the student and their characteristics. This 
review aimed to provide a synthesis of research on the effects of DGBL on STEM education. However, the study had 
limitations, as it only considered research articles published between 2018 and 2023. 

Systematic literature reviews are recognized as effective approaches in social science research for their trans- 
parency and openness to criticism through the PRISMA methodology. However, like any research method, there are 
inherent limitations in the methodology and its application that apply to this review. 

The STEM concept’s complexity and interdisciplinary nature make it difficult for researchers to maintain focus 
on the entire content. During the preparation, collection, and research of the literature, several areas that could be 
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improved were discovered. The authors focused mainly on one discipline or connected two, achieving a synergistic 
relationship (chemistry and biology; computer science and mathematics). 

Student achievements can be seen within three domains, and a challenge arises due to insufficiently detailed 
information about the teaching of STEM education supported by educational digital games. Evaluations are described 
with different focuses and methods in various depths and variations. DGBL systems are often unavailable in playable 
form, and the reproduction and verification of results could be difficult. 

Another shortcoming is that most studies assess student achievement, motivation, and engagement during 
instruction and evaluation. This study focuses only on student motivation. Other studies have positively evaluated 
DGBL in STEM education if one of the criteria is met. 

This study excludes other factors to give the most authoritative results of DGBL's effect on student motivation. 
Although the potential and possibilities of DGBL are significant for the entire education, not only STEM education, there 
are still some limitations that we must keep in mind. It is necessary to single out limitations to minimize them in practice. 

However, limitations should be seen as new topics for research and improvement. It is also necessary to monitor 
the effect of using digital games in education on students with limited financial resources who need access to digital 
devices at home. Another limitation of this study, but also an idea for future research, is the type of educational game. 
No one-size-fits-all educational game simultaneously examines the motivation of science, technology, engineering, 
and math students. Minecraft Education Edition is a good attempt, but only some of its potential has been used. 
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Abstract. The purpose of the present study 
was to examine the effect of individual and 
school average achievement on students’ 
self-concept in science. The data of the last 
four Trends in International Mathematics 
and Science Study (TIMSS) cycles were 
used, and a two-stage individual data 
meta-analysis was conducted to examine 
the relationship between achievement 

and self-concept in science. In the first 
stage, multilevel structural equation 
modeling was performed for each 

dataset. Furthermore, the effect sizes were 
synthesized via meta-analysis with a three- 
level random-effect model. In addition, 
moderation analysis was conducted for 
the year, grade, and degree of stratification 
in countries. The results suggested a 
positive and significant effect of individual 
science achievement on the self-concept of 
students in science. On the other hand, the 
relation of the average school achievement 
was negative. The results were discussed by 
considering the big-fish-little-pond effect. 
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A META-ANALYTIGC EXAMINATION 
OF THE EFFECT OF INDIVIDUAL 
AND SCHOOL-AVERAGE 
ACHIEVEMENT ON STUDENTS’ 
SELF-CONCEPT IN SCIENCE 


Nurcan Kahraman 


Introduction 


Students’ self-concept is one of the significant educational constructs 
that influence their learning outcomes (Bong & Skaalvik, 2003). Self-concept 
is about individuals’ self-beliefs about their ability in an academic domain 
(Ferla et al., 2009). Students can evaluate their abilities by using different 
references such as comparing their performances to past experiences or 
their peers’ performances (Bong & Skaalvik, 2003; Wigfield & Eccles, 2000). 
The big-fish-little-pond effect (BFLPE) theory argues that social comparison 
is an important reference point for students to evaluate their ability in an 
academic domain. In other words, according to the theory, students tend to 
compare themselves with their peers like their classmates or schoolmates and 
adopt self-beliefs based on the results of this comparison (Marsh, 2004). The 
theory underlines that those external references may influence self-concept, 
and achievement is an important external reference for students. Hence, the 
achievement of the group that students are involved in becomes an impor- 
tant predictor of students’ self-beliefs. As the average group's achievement 
increases, individuals’ self-concept decreases (e.g., Marsh et al., 2008; Seaton 
et al., 2010). For instance, Chen (2022) examined the BFLPE in mathemat- 
ics for Hong Kong students with Trends in International Mathematics and 
Science Study (TIMSS) 2019 data and supported that group achievement 
negatively affects self-concept not only for eighth graders but also for fourth 
graders. Besides, even though the BFLPE was significant for both grades, it 
was stronger for eighth graders. Marsh et al. (2014) also showed that the 
BFLPE is lower for younger students in their study in which they examined 
the BFLPE across diverse cultures. In another study, Wang (2020) examined 
the BFLPE in mathematics with a cross-national analysis, using TIMSS 2015 
data for eighth and fourth graders. Multilevel structural equation modeling 
showed that the BFLPE was significant for most participant countries for 
both grades. Fang et al. (2018) conducted a meta-analysis to synthesize the 
BFLPE. They involved 56 effect sizes from 33 studies and found a significant 
and small mean effect size (8 = —0.28). 

Nowadays, researchers tend to be interested in synthesizing the 
BFLPE from complex surveys like TIMSS with a meta-analytic perspective. 
For instance, Parker et al. (2021) used TIMSS cycles from 2003 to 2015 and 
analyzed 130 BFLPE sizes for grades four and eight with a Bayesian random 
meta-regression. According to the findings, the BFLPE was negative and sig- 
nificant except in two cases. Furthermore, there was a large, positive relation 
between the BFLPE and ability stratification. In a recent study, Compos et al. 
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(2022) examined the BFLPE in mathematics by using international large assessments (e.g., TIMSS, PISA). They used 
a two-stage parameter-based meta-analysis to estimate the effect size and confirmed the significant negative ef- 
fect of group average achievement on students’ mathematics self-concepts. Moreover, the findings suggested no 
significant differences between years in terms of the BFLPE in mathematics. In addition, Guo et al. (2018) discussed 
that most studies that explored the BFLPE focus on mathematics. More recently, the BFLPE has been examined in 
different domains like reading. According to the author’s knowledge, limited research examines the BFLPE in sci- 
ence with a meta-analytic perspective. Hence, this study aimed to contribute to the literature by adding results for 
science. To sum up, the present study aimed to examine the effect of individual and school average achievement 
on students’ self-concept in science by synthesizing the results from the last four cycles of TIMSS. 


School Stratification 


Educational systems can shape students’ school environments, which in turn influences their self-beliefs. To 
illustrate, Guo et al. (2018) examined the BFLPE in mathematics, science, and reading among 15 Organization for 
Economic Co-operation and Development (OECD) countries by considering school stratification influence. They 
used TIMSS and PIRLS 2011 databases, and intraclass correlation (ICC) as a measurement of the degree of school 
stratification. Their results suggested a negative and significant correlation between the BFLPE and ICC, which 
means that highly stratified systems feed the negative effect of school average achievement on the self-concept 
of students. In addition, comparing the magnitude of the correlation coefficients suggests that the effect in sci- 
ence is perceived as weaker than in mathematics. In a recent study, Parker et al. (2021) also examined how school 
stratification predicts the BFLPE of students. They used TIMSS datasets for 2003, 2007, 2011, and 2015 from OECD 
countries. They meta-analyzed 130 BFLPE and ICC estimates in mathematics and suggested a large relationship 
between ability stratification and the BFLPE. In other words, when students with similar abilities are encouraged to 
enter the same schools, the negative effect of group achievement average on students’ mathematics self-concept 
becomes higher than usual. On the other hand, students in selective schools may tend to feel the school accom- 
plished their success. This is assimilation and is termed the ‘reflected glory effect’ Namely, students may think that if 
they are good enough to be in this school, they can be successful at the given task (Marsh et al., 2000). For instance, 
Parker et al. (2010) examined the effects of ability grouping on students’ mathematics self-concepts and supported 
the idea that school average can have both a negative contrast effect (BFLPE) and a positive assimilation effect. 
Cultural characteristics are one of the factors affecting which of these school average achievement effects will 
dominate. For example, collectivistic countries tend to have a smaller contrast effect than individualistic countries 
(e.g., McFarland & Buehler, 1995). Another factor that determines whether the BFLPE would have a contrast effect 
or assimilation effect is socio-economic status (Parker et al., 2018). They discussed that when the BFLPE is large, 
students from a low socio-economic background tend to have a higher self-concept than expected. To sum up, 
school stratification is a critical issue that should be considered while investigating the BFLPE. However, there is little 
research about it (Parker et al., 2021). Consequently, in the current study, countries’ school stratification degree was 
treated as a moderator variable of the relation of self-concept to both school average and individual achievement. 


Research Focus 


This study aimed to examine the effect of individual and school average achievement on students’ self-concept 
in science and the effect of grade, year, and ability stratification on these relations by synthesizing the results from 
the last four cycles of TIMSS: 2007, 2011, 2015 and 2019 data. To reach the aim of the study, the following research 
questions were addressed: 


(1) To what extent is the level of achievement related to the self-concept of students in science? 

(2) Does school average science achievement relate to the science self-concept of students (BFLPE)? 

(3) Do year and grade moderate the relation of science self-concept to school average and individual 
achievement of students? 

(4) Does the degree of school stratification moderate the relation of science self-concept to school average 
and individual achievement of students? 
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Research Methodology 
Data 


TIMSS is an international exam which is carried out by the International Association for the Evaluation of 
Educational Achievement (IEA). It is administered every four years according to a schedule and focuses on fourth 
and eighth graders’ mathematics and science achievement. Additionally, it also provides data on non-achievement 
variables such as socioeconomic status, demographic characteristics, and home and class environment (Interna- 
tional Association for the Evaluation of Educational Achievement, n.d.). 

The data of TIMSS's last four cycles for both eighth-grade and fourth-grade students were used in the cur- 
rent study. For eighth-grade measures, some countries teach science in separate areas like biology, chemistry, or 
physics. Hence, they were not included in the study. Only the general science data of the survey is included in the 
analysis. Since there were no exclusion criteria except for measuring the self-concept of science in general form, 
all the available data, 345 data sets from 2007 to 2019, were used. There were 1,053,438 fourth-grade students and 
795,936 eighth-grade students. In Figure 1, The PRISMA diagram (Moher et al., 2009) presents the included and 
excluded data sets of the current study. Table 1 summarizes the number of countries for each year. 


Table 1. 
Number of Countries For Each Year 


Number of countries 
Year 
Fourth grade Eighth grade 

2007 40 34 

2011 57 34 

2015 50 32 

2019 63 30 
Figure 1 
The PRISMA diagram 


Records identified from*: 
Identifying complex survey 
(TIMSS) 

(n =395) 


Data excluded: 


Reports assessed for eligibility Science measured separately 


(n = 395) (n =48) 
CFA did not confirm the 
structure (n = 2) 


Data of included in analysis 
(n = 345) 
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Instrument and Procedures 
Science self-concept 


Three items; (i) “I usually do well in science’ (ii) Science is not one of my strengths, (iii) | learn things quickly 
in science were handled as students’ science self-concept (Wang, 2015). The confirmatory factor analysis was 
conducted for each country to examine the validity of the measurement. All the data confirmed the structure of 
the self-concept (CFI >.90, RMSEA <.08, SMR <.08) except for two datasets. Hence, the data from the exception 
countries were removed from the analyses. 


Science achievement 


TIMSS assesses students’ science achievement in four domains: biology, chemistry, physics, and earth sci- 
ence, and three cognitive areas: knowing, applying, and reasoning. Besides, achievement scores are treated as 
five plausible variables in databases. 


School stratification degree 


To assess countries’ degree of school stratification, the ICC of science achievement was calculated for each 
country. ICC shows us how the total variance in science achievement is explained by school differences. If ICC 
approaches one, it means a high school stratification system; students may be rank-ordered according to their 
achievement scores and are entered into the school. In contrast, ICC close to zero means that students are randomly 
assigned to the schools (Parker et al., 2021). 


Data Analysis 


Two-stage individual data meta-analysis was conducted to examine the relationship between achievement 
and self-concept in science. Campos et al. (2023) suggested that meta-analysis of the data with complex sampling 
designs can be complex, and statistical problems can occur, such as handling survey weights or clustered data 
structure while conducting one stage of individual participant meta-analysis. Moreover, they suggested a two-stage 
individual participant data analysis for complex survey designs. According to their proposal, the first stage comprises 
individual data analysis from the studies and the second stage makes a meta-analytic synthesis of effect sizes. 

In the current study, multilevel structural equation modeling analysis was performed for each of the data sets 
for the first stage of meta-analysis. Since achievement was assessed via five plausible variables, each analysis was 
run five times and combined for the data estimates. The quadratic component of science achievement was also 
included in the model at Level 1. At Level 2, the school average of science achievement was obtained by rounding 
the grand mean. Equation 1 and 2 represents a summary of the model. 


scwij = Bwithinachwij+ B2(achij)? + éij (1) 
scbj = Bbetweenachbj+ 6j (2) 
swij: within level self-concept, sbj: between level self-concept, achij: student science achievement in group j, 
achbj: the average science achievement for the group. 


The R programming language and Mplus software were used for data analysis. The packages of EdSurvey, 
MplusAutomation, tidyverse, and rlang were installed in R. The codes were adopted from Wang (2020). The stan- 
dardized effect sizes were calculated by using equations 3 and 4 (Wang & Bergin, 2017). 


ES=2 x B x SD(science)/SD(scij) 
BFLPE ES=2x (Bbetween- Bwithin) x SD(science)/SD(scij) 
8: unstandardized regression coefficient, SD: standard deviation 


— 
 W 
ex 
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In Step 2, a meta-analysis was performed for (1) science achievement within the level, and (2) the BFLPE sepa- 
rately. Three-level random-effects models with the sampling variance for each effect size (Level 1), variation effect 
sizes within countries (Level 2), and between countries (Level 3). The analysis was conducted using the metafor 
package in R. The codes were adopted from Campos et al. (2023). 


Research Results 
Effect Size 


Before calculating the pooled effect size, Cochran’s Q was used to examine the heterogeneity. It is the classic 
method for assessing heterogeneity in meta-analyses. Using a chi-squared distribution, it calculates the prob- 
ability that, if significant, indicates greater diversity between studies than within subjects within a single study 
(Higgins & Green, 2008). The results of Cochran's Q statistics suggested heterogeneity among studies for the BFLPE 
(Q [343] = 2195.58, p <.001) and the effect of within-level achievement (Q[343] = 9202.89, p <.001). According to 
the I? values of the variables, the magnitude of the heterogeneity was 93.51% for the effect of science achievement 
and 85.56% for the BFLPE. Hence the heterogeneity is high for both variables (Borenstein et al., 2009). The three- 
level random-effect model suggested a significant negative BFLPE (ji = -.27) and a significant positive achievement 
effect (ji = .60). The findings are presented in Table 2. 


Table 2 
Average Effect Sizes and Heterogeneity 


i (se) 95%Cl a? o? P... P. P a QE (af) 
BFLPE -27(.01) — -.29, -.25 19 Od 82.57 299 85.56 2195.58 (353) 
Within-level linear effect 60(.01) 58, 63 a 03 93.54 234 95.56 9202.89 (353) 


fi= Weighted average effect siz, 0”, = Variation between effect sizes within the countries, o? =Between-country variation, ? ,.,  ,, and are the cor- 
responding heterogeneity indices, QE= test fot heterogenity 


Analysis of Moderator Variables 


After calculating the general effect size, the grade was added to the model as a moderator of the variables. 
Q statistics showed that the moderator is significant for both the BFLPE (Q [1] = 38.86, p < .01) and within-level 
achievement (Q [1] = 73.36, p < .01). In other words, there was a significant difference between fourth and eighth 
graders in terms of the effect of not only individual achievement but also school average achievement on science 
self-concept. According to the results, the relationship between achievement and self-concept of eighth-grade 
students is larger than the relationship between achievement and self-concept of fourth-grade students. In an- 
other model, the year was considered as a moderator of the outcome variables. According to the findings, the year 
was not a significant predictor of the BFLPE. It was significant for science achievement. The highest score was for 
2007 whereas the lowest score was for 2019. Finally, the ICC of science achievement was added to the model as 
a moderator to examine the stratification effect. According to the results, ICC was only significant for within-level 
achievement. The findings of each moderator analysis are presented in Table 3. 


Table 3 
The Findings of Moderator Analyses 


Moderator Icc Year Grade 
BFLPE Within level BFLPE Within level BFLPE Within-level 
linear effect linear effect linear effect 
fi (se) -.24 (.03)* 54 (.02)* -.29 (.02)* 70 (.02)* -.22 (.01)* 53 (.01)* 
95 % Cl -.29, -.19 48, .59 -.34, -.24 .65, .75 -.25, -.18 50, .56 
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Moderator ICC Year Grade 

BFLPE —neareftct «== BFLPE—ngaretect —BFEPE inert 
ICC -.03 .07* - - - - 
2011 [95 % Cl] - - -.03 [-.01, .03] -.10 [-.16, -.04]* - - 
2015 [95 % Cl] - - 04 [-.03, .10] -.13 [-.20, -.07]* - 
2019 [95 % Cl] - : 05 [-.01, .11] -.16 [-.22, -.09]* : - 
Grade 8 [95 % Cl] - ; -14[-.18,-.09]* —.19[.14, 23)" 
e., 19 21 18 20 18 19 
o7. 04 .04 04 .04 .04 03 
Pes 82.47 93.09 80.92 92.31 79.80 92.45 
P 3.07 2.68 3.41 2.92 3.35 2.44 
Pas 84.54 95.77 84.33 95.24 83.16 94.89 
QE (df) 2180.92 (352)* 8864.79 (352) 2053.54 (352)* 8478.97 (352)* 1970.32 (352)* 6655.12 (352)* 
QM (df) 1.32 (1) 6.05 (1)* 3.49 (3) 27.67 (3)* 38. 86 (1)* 73.36 (1)* 


fi= Weighted average effect siz, 0,,,? = Variation between effect sizes within the countries, 0,2 = Between-country variation, /,.”, 1’, and are 


the corresponding heterogeneity indices, QE= test fot heterogenity, QM = Test of Moderator, *p <.07 
Discussion 


The current study aimed to synthesize the effect of individual and group average achievement on the self- 
concept of students in science. To achieve this aim, the last four cycles of TIMSS data for grade four and grade eight 
were used. Concerning Research Question 1 (To what extent is the level of achievement related to the self-concept 
of middle school students in science?), results suggested a significant and positive linear effect on students’ science 
self-concept within the level. The magnitude of the pooled effect size was 0.54 and the heterogeneity of the effect 
size was extremely high within countries (/? = 93.09%). Contrariwise, there was little heterogeneity between countries 
(P = 2.68 %). Research Question 2 was related to the BFLPE. Results suggested that school average achievement 
has a significant and negative effect on students’ science self-concept. In other words, the findings supported the 
BFLPE theory for science. According to three-level random-effects models, although most of the heterogeneity 
lies within countries (7 = 82.47%), heterogeneity also exists between countries (/? = 3.07%). In respect thereof, it 
will be useful to mention the findings of the first stage. According to the multilevel structural equation modeling 
analyses, the achievement was positively significant for science self-concept for nearly all countries for each grade 
within the level, and the school achievement average (BFLPE) was negatively related to the science self-concept of 
students for most of the countries. This finding is parallel to previous studies’ results. For instance, Guo et al. (2022) 
examined the BFLPE in three different domains, mathematics, science and reading, across 15 countries and found 
that the between-country variance is too small. Moreover, they explored the relationship between the degree of 
ability stratification and the BFLPE at the country level and suggested that the relationship for mathematics is 
stronger than the relationship for science. Supporting this idea, in a recent study, Campos et al. (2023) synthesize 
the size of the BFLPE in mathematics with three-level random-effects models and suggest that most of the het- 
erogeneity lies between countries (/? = 78%). Hence in a further study, the BFLPE can be examined with individual 
data meta-analysis for both science and mathematics over the same data sets. 

In addition, moderator analyses were conducted to understand the role of year and grade on the heterogene- 
ity of the effect of both individual and school average achievements on students’ self-concept. Findings supported 
non-significant heterogeneity among cycles of TIMSS regarding the BFLPE. Namely, the year did not contribute 
significantly to the heterogeneity of the school average effect size. On the other hand, it was a significant modera- 
tor of the linear effect of science achievement. While the lowest effect size was for 2019, the highest one was for 
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2007. Furthermore, the grade was a significant predictor for the BFLPE and within-level achievement; eighth-grade 
students tend to feel the effects more strongly than fourth-grade students. This was an expected finding because 
previous studies also offer a smaller BFLPE for younger students (e.g., Chen, 2022; Marsh, 2014). 

Another aim of the current study was to examine whether the degree of school stratification controls the 
relationship between achievement and self-concept in science. Therefore, the ICC of science achievement for each 
country was considered as a moderator for the effect sizes both for school average achievement (BFLPE) and indi- 
vidual achievement. According to the results, there was no significant moderation impact on the BFLPE. This was a 
surprising finding since it is expected that selective educational systems tend to cause a negative BFLPE on students’ 
self-concept. In addition, previous results suggested negative relations between the degree of stratification and the 
BFLPE. For instance, in their study, Parker et al. (2021) meta-analyzed 28 OECD countries’ data in mathematics and 
suggested a large and negative significant relation between ICC and the BFLPE. In another study, Guo et al. (2018) 
examined this relation not only for mathematics but also for science and reading and confirmed the significant and 
negative impact of school stratification on the BFLPE of students. They also demonstrated that the negative effect 
is larger in mathematics compared to science. However, they examined this effect only in OECD countries. These 
countries mostly show individualistic characteristics, and individualistic countries tend to have a larger negative 
effect of school average achievement on the self-concept of students than collectivistic cultures (e.g., McFarland 
& Buehler, 1995). On the other hand, the data of the current study involves not only OECD countries, but all TIMSS 
participant countries involved in the analyses, so in some countries, a positive assimilation effect of the BFLPE 
can occur. For instance, in the reflected glory effect, the student can perceive the group success as a reference to 
evaluate their ability in an academic domain (Marsh et al., 2000). Therefore, these cultural or economical differences 
may cause a drop in the significance of school stratification. In a further study, the school stratification effect can 
be examined by considering the cultural differences among countries. 

To sum up, the effects of both individual and school average achievement were meta-analyzed in this study. 
The last four cycles of TIMSS data were searched, and 355 data were involved in the analysis. This study makes 
several contributions. Firstly, considering the relationship between self-concept and other educational outcomes 
(e.g., Sudrez-Alvarez et al., 2014), it is important to examine the factors that influence students’ self-concepts, of 
which achievement is a prominent predictor. Hence, this study examined the effects of achievement in two forms: 
the linear individual effect within the level, and the school average form (BFLPE). Next, although the BFLPE is mainly 
explored for mathematics in the relevant literature (Guo et al., 2018), the magnitude of the effects can vary with 
the academic domain. To the author’s knowledge, this is the first study that synthesizes the BFLPE with individual 
participant data meta-analysis in science. 


Conclusions and Implications 


The study presented an insightful examination into the influence of individual and school-average achievement 
on students’ self-concept in science, using data from multiple cycles of TIMSS. The results illuminated the nuanced 
relationship between academic achievement and self-concept, particularly emphasizing the significant impact of 
both individual science achievement and school-average achievement on students’ perceptions of themselves in 
the scientific domain. The results concerning the BFLPE in science which are consistent with previous research in 
mathematics and provide a comparative perspective across academic domains. The study’s examination of the 
BFLPE in science education, an area with limited prior meta-analytic investigation, adds depth to the literature. 
Recognizing the impact of group achievement on individual self-beliefs could lead to tailored interventions or 
teaching methods to counteract negative effects. 

In addition, the study examined the moderating roles of grade, year, and school stratification, providing insights 
into how these factors influence the associations between achievement and self-concept. The study highlights 
the influence of grade level on the relationship between achievement and self-concept. Understanding that older 
students may be more affected by group performance could lead to age-specific interventions aimed at increasing 
self-confidence, especially in subjects where the BFLPE is more apparent. 
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Abstract. Smartphones and tablets have 
permeated various aspects of life. This 
study explores the differences in wishes 
between students, parents, and teachers 
regarding the use of smartphones and 
tablets in biology classes in the upcoming 
school year. An online questionnaire 

was used for the study, which provided 
eight different scenarios for the use of 
smartphones for teaching purposes. The 
data were collected from 934 participants, 
including 465 students, 282 parents, 

and 188 biology teachers from various 
Slovenian lower secondary schools. The 
principal component analysis revealed 
the unidimensional structure of the 
instrument, explaining 59.7% of the 
variance (alpha =.91). The results showed 
that the use of smartphones and tablets 
for distance learning, teaching purposes, 
schoolwork and homework is generally 
desirable. There was less consensus on 
their use for laboratory and field work, 
evaluation of knowledge, and biology 
lessons. The main finding was that the 
differences between the groups were small 
or even negligible in terms of effect sizes. 
Statistically significant differences were 
found between the focus groups, with 
students and teachers expressing greater 
agreement than parents. These findings 
emphasize the importance of addressing 
parents’ concerns and understanding the 
perspectives of stakeholders in order to 
effectively integrate smartphones and 
tablets into the classroom. 
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OF STUDENTS, TEACHERS, AND 
PARENTS ON INTEGRATION OF 
SMARTPHONES AND TABLETS 
IN BIOLOGY LESSONS 


Vida Lang, 
Andrej Sorgo 


Introduction 


The integration of information and communication technologies into 
society has become increasingly important, and several seminal strategic 
documents promote even greater digitalization, including the education 
sector (European Commission, 2021; OECD, 2019). One area of technologi- 
cal advancement that has attracted much attention recently is the use of 
smartphones and tablets in the classroom (Anshari et al., 2017; Nikolopoulou, 
2020). While their use as communication tools and powerful pocket-sized 
computers has become ubiquitous in private lives (Boyd, 2014), influencing 
both utilitarian and hedonistic aspects of life (Akdim et al., 2022), their use in 
education is controversial at best (Anshari et al., 2017; Flanigan & Babchuk, 
2022). As these devices, according to the references (Hartley & Andujar2022), 
offer a range of opportunities to enhance teaching and learning experiences, 
provide access to educational resources, promote collaboration and increase 
engagement, smartphones and tablets have therefore become, at least po- 
tentially, a learning tool with great potential for formal and informal learning 
(Fu & Hwang, 2018; Gikas & Grant, 2013: Kacet! & Klimova, 2019; Longman 
& Younie, 2021; Qureshi et al., 2020; Statti & Villegas, 2020). Although the 
integration of learning with mobile technologies in education has been ex- 
plored by many, adoption and integration in lectures are still limited due to 
various barriers (Criollo et al., 2021; Nikolopoulou, 2020; Nikolopoulou, 2021; 
Sanchez-Prieto et al., 2019; Whyley, 2018). For example, smartphones/cell 
phones in the classroom are seen more as a distraction rather than a learning 
aid (Anshari et al., 2017; Beeri & Horowitz, 2020), which is why they may even 
be banned in schools (Montag & Elhai, 2023) due to problematic smartphone 
use (Busch & McCarthy, 2021) or their impact on health and mental health 
(Abi-Jaoude et al., 2020; Brodersen et al., 2022; Odgers, 2018). While a ban 
may prevent some problems related to the negative use of smartphones, the 
question is who should teach students how to use them wisely. Unguided 
use for self-education, based on the assumption that students are digital 
natives because of their intensive use (Prensky, 2001), has failed (Lang et al., 
2024). Furthermore, Dolenc and Sorgo (2020) have shown that students who 
spend more time using digital devices may actually exhibit lower levels of 
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information literacy and that the skills and cognitive abilities students need for academic success are negatively 
affected by excessive phone use (Sunday et al., 2021). 

The guiding premise of our work was that schools may not have the power to restrict smartphone use outside 
of school grounds, but they can at least potentially increase the quality of digital work by providing learning expe- 
riences and teaching students how to use mobile devices wisely. The starting point was the paradoxical situation 
that a complete ban on smartphones in schools could be counterproductive because students are not taught how 
to use smartphones intelligently in the name of preventing their potentially and actually harmful effects, even if 
the benefits can be recognized as a lifelong skill. Examples of such applications in the life sciences include the use 
of smartphones in health and environmental monitoring, biodiversity documentation and collaboration in citizen 
science projects, among many other possibilities. 

Recent research has investigated the potential applications of smartphones and tablets in biology lessons (Lang 
& Sorgo, 2023). Many possibilities have been tested with prospective science teachers as well as with primary and 
secondary school students. The collective observations that emerge from both our experience and the comprehen- 
sive study of established practices show that smartphones and tablets are useful in biology education. In addition 
to their basic role as a source of communication and information, these devices can also serve multifunctional 
purposes, e.g. as data loggers, microscopes, and identification keys (Lang & Sorgo, 2023). As school biology also 
covers topics such as health and the environment, some applications whose primary intention is not education, 
but health monitoring can also be included (Ernsting et al., 2017; Mosa et al., 2012). In summary, however, accord- 
ing to an analogy used by many, smartphones and tablets function like a “Swiss army knife” (Adalar, 2021), which 
can be useful and helpful but cannot replace a toolbox of professionally developed devices for teaching biology. 

The successful use of smartphones and tablets in biology classrooms not only faces tensions between the 
content to be taught, the technology to be used, and the pedagogy to be applied in a given context (Mishra, 2019), 
but also depends on understanding the preferences and perspectives of teachers, parents, and students, who are 
key stakeholders in educational processes that link school and home (Gao et al., 2017; Matteucci & Helker, 2018). 
The differences between views, expectations, and wishes regarding the use of technology, as well as the pedagogy 
used and possible side effects, can be seen as an important issue to explore. The key players in this process are the 
teachers because they hold the key to technology in the classroom. However, parents can be seen as promoters 
and suppressors of the educational use of smartphones/tablets (Hadad et al., 2020). On the other hand, teachers’ 
power ends at the front door, where they can influence smartphone use by giving students homework that involves 
smart devices. This situation can lead to tensions between the three partners, as only the children are “full-time 
employers” in both worlds. Therefore, it would be wise to design the use of smart technologies in a way that aligns 
with the wishes and expectations of all three partners in the educational process. Despite their importance, our 
search did not reveal any studies that investigated the different perspectives of students, teachers, and parents 
regarding the use of smartphones in (science) biology lessons using a standardized quantitative survey instrument 
across all participant groups. The research gap to be filled was, therefore, to investigate what students, teachers, 
and parents wish to use smartphones and tablets for various purposes in class and for schoolwork in the coming 
school year. Knowing wishes can be important because they can be actualized in actual educational use in the 
sense of theories such as the Theory of Reasoned Action (Ajzen, 1980), the Technology Acceptance Model (Davis, 
1989; Davis et al., 1992) and UTAUT (Venkatesh et al., 2003). The aim was to identify potential inequalities that could 
hinder the integration of an effective multipurpose tool in (science) biology education. This tool should facilitate 
the transformation of skills acquired in biology classes into lifelong learning opportunities. 


Research Aim and Research Questions 


The aim of the research was to comprehensively understand the wishes of students, parents, and teachers 
regarding the use of smartphones and tablets for educational purposes, with a focus on biology education. At this 
point, it is appropriate to point out a limitation of the study resulting from the guaranteed anonymity of the survey. 
This limitation meant that it was not possible to compare the wishes of specific teachers, their students, and parents. 

Therefore, a series of research questions were posed that relate to the topics shown in Table 1. All research 
questions can be interpreted in terms of the hypothesis that there are statistically significant differences in each 
of the listed items concerning the expected use of smartphones and tablets in the classroom and at home in the 
context of biology (science) lessons. The following research questions were formulated: 

RQ1: How often do students, teachers, and parents wish to use smartphones and tablets for various 
teaching purposes in biology lessons in the coming school year? 
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RQ2: How do differences in the wishes of students, parents and teachers differ regarding the use of 
smartphones and tablets for various teaching purposes in biology lessons in the coming school 
year? 

RQ3: What significant differences are there between the wishes of students, parents, and teachers 
regarding the use of smartphones in biology lessons? 

RQ4: To what extent do the wishes of students, parents and teachers agree when it comes to the 
integration of smartphones in biology lessons? 


Research Methodology 
General Background 


For this study, a survey strategy was implemented with the use of an online questionnaire. The aim of the 
questionnaire was to collect information about the differences in wishes of students, parents, and teachers regard- 
ing the use of smartphones and tablets in biology lessons. Data collection began in January 2023 and ended in 
March 2023 at lower secondary schools throughout Slovenia. 

It is imperative to acknowledge an inherent limitation of the study— the guaranteed anonymity of the survey. 
This limitation restricted the ability to match specific teachers with their corresponding students and parents. 


Sample 


The survey instrument in the form of an online questionnaire based on the 1ka platform (www.1ka.si) was made 
available through various channels, online social media, contacts with schools and individual teachers. Data collection 
started in January 2023 and ended in March 2023. The survey was conducted in accordance with Slovenian guidelines 
and regulations for educational research, provided that no personal or sensitive data were collected. As the survey 
was completely anonymous and voluntary, respondents’ decision to take part in the survey was not influenced by 
social media or peer relationships. However, this strategy made it possible to assume that at least potentially every 
member of the target population of interest, which was teachers, parents, and lower secondary school students in 
the last grades (8th and 9th) of the nine-year compulsory Slovenian school, could provide answers to the survey. 

The exact population number of teachers, parents, and lower secondary school students in the last grades (8 and 
9") of the nine-year Slovenian compulsory school can only be roughly estimated and is around 20,000 for a generation 
of students, approximately two times more for parents, and about 150 biology teachers teaching each generation. 
The response rate was as follows: 3234 visits to the survey homepage, 2139 (66%) continued to the first page with 
an explanation of the aims of the survey, 1967 (61%) gave partial responses, and 1041 (32%) gave all responses with 
irregularly positioned missing data. Since the aim was to analyze the records of their wishes to use smartphones and 
tablets in class and for schoolwork in the coming school year (codes Q15a — q15h in tables and figures) — selected 
were only those who answered this question completely. Therefore, the research sample includes 934 participants. 
Of these, 465 (49.8%) were students from various Slovenian lower secondary schools (the last two final grades) and 
the first grade of upper secondary school; 281 (30.1%) were parents, and 188 (20.1%) were teachers. The decision to 
collect data also from students in the first grade of upper secondary school was practical, as their experiences can 
be matched to a range of teachers and schools. The population of participating students consisted of 19.1% Grade 8 
and 9 lower secondary students and 80.9% Grade 1 students. The students described themselves as male students 
(27.7%) and female students (67.8%), while the remainder (4.5%) did not wish to provide gender information. 

In addition to basic information about the parents’ status, they answered whether they were currently involved 
in education. The majority, or 80.8% of respondents, mentioned that they were not currently employed in the field 
of education, while the remaining percentage (19.2%) confirmed that they were employed in education. Participat- 
ing parents described themselves as male (23.5%) and female (76.1%), while the rest (0.4%) did not want to provide 
gender information. 

The population of participating teachers consisted of lower secondary biology teachers (47.3%), upper second- 
ary biology teachers (38.6%) and teachers of other subjects (14.1%). When examining the differences between the 
three groups of teachers, no statistically significant differences were found. Therefore, they were treated together as 
a single group of teachers. The participating teachers described themselves as male (9.2%) and female (90.2%), the 
rest (0.6%) did not want to give gender information. 

One limitation of the study could be the representativeness of the sample. The biggest limitation in data collec- 
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tion is self-selection, which can lead to distorted answers. The other problem is the lack of responses from the unseen 
majority of students, parents, and educators. For them, we can only speculate that their answers match those of the 
respondents. However, it is impossible to correct this potential flaw in the study design. 


Instrument 


A structured questionnaire was used as the primary tool for data collection. The instrument of interest for the 
study took the form of a table in which the introductory statement was followed by nine items. Students, teachers, 
and parents were asked about their wishes regarding the use of smartphones and tablets for various purposes in 
class and for schoolwork in the coming school year. Theoretically, wishes regarding the use of technology can be 
seen as a correlate to behavioral intentions, which may or may not predict actual behavior depending on facilitating 
conditions as defined by theories such as the Theory of Reasoned Action (Ajzen, 1980), the Technology Acceptance 
Model (Davis, 1989; Davis et al., 1992) and UTAUT (Venkatesh et al., 2003). 

The instrument includes 9 items (see Table 1) that participants were asked: “We would like to know if you would 
like the smartphones and tablets to be used for the following purposes in the coming school year.’ The response 
format was a 6-point scale of 0 (never), 1 (very rarely), 2 (rarely), 3 (occasionally), 4 (often), 5 (very often). In the 
statistical analyses, the values were converted from 1 (never) to 6 (very often). Later in analyses, Item 9 (other) was 
excluded, and the results of this item are not reported. 

The reliability of the questionnaire was assessed by calculating Cronbach's alpha, which resulted in a high value 
of .907. Content validity was ensured through consultations with the experts in the field and by piloting previous 
versions of the questionnaire (Lang & Sorgo, 2022; Lang & Sorgo, 2023). The convergent validity of the construct was 
assessed by checking the component loadings, which were calculated by applying principal component analysis. 
Allitems have a loading of over .6 (see Table 1). 


Data Analysis 


Each research variable was examined for central tendencies, dispersion, and normality distribution. Based on 
the data, non-parametric statistics were chosen for further testing. A polychoric correlation matrix was calculated, 
and a Gaussian graphical model was used to graphically represent the polychoric correlation matrix. By summation 
of eight items, it was possible to identify the respondents who did not wish to use smartphones and tablets (sum 
= 8) in any form in the next year, as well as those who wished to use smartphones very often in all listed (sum = 
48). Because of the non-linear scale, sums between the extremes were not further explored. 

Principal component analysis (PCA) with direct oblimin rotation was conducted to assess the dimensional- 
ity of the instrument. The reliability of the components that emerged from the PCA analysis was assessed using 
Cronbach's alpha coefficient, and the value of .7 was chosen as the cut-off value for continuing the analyses. To 
further explore the structure of the instrument, a CFA analysis was performed. Due to the ordinal nature of the 
data, the DWLS (Diagonally Weighted Least Squares) method was applied to explore the predicted dimensional 
structure of the matrix. 

To assess the statistical significance of the differences between the participant groups (students, parents, and 
teachers), the non-parametric Kruskal Wallis Test was applied, and the €” (epsilon squared) values are reported as 
a measure of the effect sizes. The criterion for demonstrating significance is the p-value as p < .05 and the effect 
sizes are interpreted as: .00 < .01 — negligible; .01 < .04 - weak; .04 < .16 — moderate; .16 < .36 — relatively strong; 
.36 < .64 - strong; .64 < 1.00 - very strong (Rea & Parker, 2014). Since three groups were considered, pairwise com- 
parisons were made using the Dwass-Steel-Critchlow-Fligner statistic. Statistical analyses were conducted using 
the open-source statistical program jamovi, 2.3 (The jamovi project, 2022). CFA was performed using the tools of 
the Structural Equation Modelling package (SEM module) in jamovi. 


Research Results 
Based on the research conducted, general results were presented on the students, parents, and teachers’ 


wishes to use smartphones and Tablets for various purposes in class and for schoolwork in the coming School 
year. 
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Table 1 
Measures of Central Tendencies and PCA Loadings of Wishes to Use Smartphones and Tablets for Various Purposes in Class 
and for Schoolwork in the Coming School Year (n = 934) 


Codes Items Me Mo PCA 
Q15h To participate in distance learning (video conferences). 5 6 64 
Q15a For teaching purposes in any school subject. 4 4 87 
Q15d For schoolwork at school. 4 4 86 
Q15c For homework. 4 4 19 
Q15b In biology class. 4 4 83 
Q15e For field work. 4 4 18 
Q15g To test knowledge. 4 4 15 
Q15f For laboratory work. 4 4 19 

Variance 59.7 
Eigenvalue 5.37 
Cronbach's alpha 907 


Note. KMO = .920; x2 = 2276, df = 36; p <.001, PCA = component loadings. 


All combined results of students, parents, and teachers’ wishes (Table 1) on smartphone use in the biology 
classroom are above the median value (4) of the scale. At the high end of the scale (median = 5; mode = 6), where 
participants consider the use of smartphones and tablets in the classroom is the use “For distance learning par- 
ticipation (video conferencing)”. For all other items, except Other, the median and mode values were the same (4), 
which means ‘occasionally’ in absolute terms. The item ‘Other’ (Q15i) was not included in the follow-up analyses. 
Even though the differences are small, use in biology lessons, fieldwork, tests, and lab work are at the end of the 
series, which shows the tendency that smartphones are most likely seen as a general means of communication 
and access to information, rather than as a more specialized educational tool. 

In order to gain further insights, the answers of the participants who answered all the questions (n= 903) were 
summarized. It turned out that only 17 students (1.9%) did not wish to use smartphones for educational purposes 
in the next year. At the other end of the scale, 41 people from all three groups (4.5%) wished to use smartphones 
very frequently in all the situations listed. Due to the non-linear scaling, the results between the minimum and 
maximum were not interpreted. 

The reliability of the instrument was high, as shown by Cronbach's alpha value of .907. After PCA, only one 
component was identified that explained 59.7% of the variance with an eigenvalue of 5.37. This indicates the uni- 
dimensionality of the instrument designed to assess wishes to use smartphones and tablets in the coming school 
year. Based on this result, the combination of items could be used as a latent construct in follow-up studies, if such 
studies are planned. 

Additional insight into the relationships between the items was gained by using the polychoric correlation as 
a measure of effect size. The polychoric correlation coefficient, a measure of the relationship between ordinal vari- 
ables, was used. The values were listed in Table 2 and plotted in Figure 1 to illustrate the Gaussian graphical model. 


Table 2 
Polychoric Correlation Coefficients of the Relationship between Variables 


Code Q15a Q15b Q15c Q15d Q15e Q15f Q15g Q15h 
Q15a 1 

Q15b 19 1 

Q15c 12 63 1 

Q15d 18 14 1B 1 

Q15e 64 63 59 63 1 

Q15f 64 68 7 67 16 1 
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Code Q15a Q15b Q15c Q15d Q15e Q15f Q15g Q15h 
Q15g 65 65 60 64 54 56 1 
Q15h 59 44 56 50 54 51 50 1 


Note. The values above .7 are printed in bold. For an explanation of the codes, see Table 1. 


By analyzing the polychoric correlation matrix it was realized that only wishes to use smartphones and tablets 
in biology class correlate with participation in distance learning below the 0.5 levels interpreted as a threshold for 
moderate correlation. The pattern most probably indicates the view that biology cannot be completely taught 
online because of many constraints, such as the use of practical and hands-on activities, as well as demonstrations 
and field work. 

All other correlations can be interpreted as moderate or, in three cases, even as strong, exceeding the .75 level. 
Four items exploring the wishes to use smartphones and tablets for teaching purposes (Q15a — Q15d) correlate 
above the .7 levels making a cluster. Additionally, the correlation coefficient indicates a moderate to strong correla- 
tion between the wish to use a smartphone or tablet for laboratory and field work (0.76), allowing the formation of 
one variable instead of two. The strength of the relations is presented by the Gaussian Graphical Model in Figure 1. 

The model was further scrutinized by applying the CFA using the DWLS method. Constraining the variables 
Q15e and Q15f (laboratory and fieldwork) resulted in the following model fit: 7 = 71.1, df= 19, p < .001; TLI = 0.997; 
SRMR = 0.031; RMSEA = 0.055 (CI95 = 0.042 — 0.069); AVE = 0.568. Standardized loadings ranged from .57 (Q15h) 
to .90 (Q15a). 


Figure 1 
Gaussian Graphical Model Representing the Polychoric Correlation Matrix 
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Note. For an explanation of the codes, see Table 1. The figure was created with jamovi 3.2. 


Differences Between the Wishes of Students, Parents, and Teachers 


In Table 3, the medians, modes and results of the nonparametric Kruskal-Walis test used to determine if there 
were significant differences between the focus groups regarding the wishes to use smartphones and tablets in 
school and for schoolwork, and accompanying effect sizes are provided. The key finding was that the differences 
between the three groups are minor or even negligible in terms of effect sizes. In order to gain further insights, 
pairwise comparisons were carried out. Only p - p-values are presented in Table 3. Even if differences are small there 
exist some patterns when individual items are pairwise compared. Statistically significant differences between the 
three focus groups occur in six statements (see Table 3), as follows: 
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a) For teaching purposes in any school subject (x? = 10.38; df = 2; p < .05), where students and teachers 
are statistically more in agreement with the item than parents. 
b) In biology class (7? = 11.17; df= 2; p < .05), where biology teachers are statistically more in agreement 
with the item than students and parents. 
c) Forschoolwork at school (9 = 11.16; df = 2; p < .05), where students and biology teachers are statisti- 
cally more in agreement with the item than parents. 
d) Forlaboratory work (7 = 9.39; df= 2; p <.05), where biology teachers are statistically more in agreement 
with the item than students and parents. 
e) Totest your knowledge (7? = 13.46; df= 2; p < .05), where biology teachers and students are statistically 
more in agreement with the item than students and parents. 
f) To participate in distance learning (video conferences) (2 = 13.64; df = 2; p < .05), where parents are 
statistically more in agreement with the item than students and biology teachers. 
g) For field work, after reviewing results pairwise, find that all three focus groups share similar thoughts 
about the use of smartphones for fieldwork. 
In summary, it can be said that the disagreement in wishes between students and parents is the greatest 
(4 points), followed by the disagreement in wishes between parents and teachers (3 points) and teachers and 
students (2 points). 


Table 3 
Differences between Students, Parents, and Teachers (N = 934; n students = 465; n parents = 281; n teachers = 188) Regarding 
Their Wishes to use Smartphones and Tablets for Various Purposes in Class and for Schoolwork in the Coming School Year 


Students Parents Teachers Kruskal-Walis test p- DSCF 


Stud- Stud- Par- 


2 2 
Code Item Me Mo Me Mo Me Mo xX df p & par teach Teach 


Q15h — To participate in distance learning 5 6 5 5 4 4 1363 2 001 0.01 898 .002 .003 


(video conferences). 
Q15a_—_- For teaching purposes in any 4 4 4 4 4 4 1038 2 006 0.01 .005 ATT .200 
school subject. 
Q15b In biology class. 4 4 4 4 4 4 117 2 004 0.01 199 114 001 
Q15c For homework. 4 4 4 4 4 4 6283 2 071 0.01 .086 .990 153 
Q15d For schoolwork at school. 4 4 4 4 4 4 1.16 2 .004 0.01 .003 457 173 
Q15e For field work. 4 4 4 4 4 4 080 2 671 855e4 .905 811 642 
Q15f For laboratory work. 4 4 4 4 4 4 939 2 .009 0.01 .034 033 .990 
Q15g ‘To test knowledge. 4 4 4 4 4 4 1346 2 .001 0.01 .002 883 015 


Note. The values of p < .05 are printed in bold; Me- Median, Mo- mode; p DSCH = p-values of the Dwass-Steel-Critchlow-Fligner 
Pairwise comparisons. 


Discussion 


Looking back at the results of the study, in which students, parents and teachers were asked about their wishes to 
use smartphones and tablets for educational purposes in the next school year, it can be concluded that smartphones 
have already become an integral part of their educational experience and will continue to be so. It is evident that the 
increasing use of digital technologies, including smartphones, for online distance learning purposes is a consequence 
of the COVID-19 pandemic (Misirli & Ergulec, 2021) and is creating new contexts of use. The results show that all the 
smartphone use suggestions listed in Table 1 were above the median (occasionally) of the scale, with anchors at the never 
and very often values. Only a minority of respondents (1.9 %) do not wish to use smartphones for educational purposes 
in the next year. Interestingly enough, there is a finding that all of them were students, violating the assumption of 
students as digital natives and teachers as digital immigrants (Prensky 2001). This reasoning stems from an assumption 
that information literate students will use digital tools for knowledge enrichment even if they are not directly instructed 
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to do it by teachers. In addition, it is known that smartphones, outside their role as communication tools, are among 
adolescents mainly used for entertainment rather than for education and self-education (Akdim et al., 2022; Boyd, 2014; 
Dolenc & Sorgo, 2020). 

Respondents in all three groups wish to use smartphones and tablets mainly for video conferencing, homework, and 
schoolwork in various subjects, with wishes to use smartphones specifically in biology classes lagging somewhat behind 
general use. This result can be interpreted to mean that the practical skills acquired in laboratory and field exercises, as 
well as the 3D models and specimens used in biology classes, cannot simply be replaced by these pocket-sized devices. 
At the end of the predicted uses of smartphones that accompany laboratory and field work is also the assessment and 
testing of knowledge. The open-ended stays question of why the wish for smartphones to be used as an alternative to 
questioning in formative assessment during a lesson (Conejo et al. 2016) is relatively low. From the teacher's point of view, 
this result might indicate that preparing online tests and assignments involves a lot of work. An additional explanation 
may be whether being online is an obstructive necessity when assignments are part of the lesson and whether this opens 
the door for online misbehavior by students. A possible explanation is also that homework based on testing is rare in 
biology compared to, for example, mathematics or foreign languages. 

The results of the polychoric correlation analysis provide insights into the relationships between different variables 
related to wishes for smartphone and tablet use in educational settings. The analysis highlights several strong positive 
correlations indicating, for example, a moderate to strong relationship between the wishes to use smartphones and tablets 
for teaching other school subjects and for teaching biology. This suggests that individuals who express a preference for 
using these devices for educational purposes are more likely to desire their use specifically in biology classes, as well as 
for schoolwork within the school environment and for completing homework assignments. Coefficients for these cor- 
relations ranged from 0.72 to 0.79. Wishes to use smartphones and tablets for schoolwork in biology, both in the school 
setting and for homework, also have strong correlations. This indicates that individuals who express a propensity to use 
these devices for school-related tasks are particularly interested in using them for biology-related tasks. The moderate to 
strong correlation of 0.76 between the intention to use smartphones or tablets for laboratory and field work indicates a 
potentially positive relationship. Thus, Lang and Sorgo (2022) could indicate that students who express a stronger intention 
to use smartphones or tablets for practical activities such as laboratory and field work find applications such as Pl@ntnet 
and the similar application iNaturalist (Rode & Torkar, 2023; Schmidthaler et al., 2023) useful for species identification. The 
correlation coefficient of 0.76 indicates a robust relationship, suggesting that as the intention to use these devices for 
practical work increases, so does the likelihood that they will see value in applications designed for educational purposes. 

In addition, the analysis shows a moderate to strong correlation (coefficient of 0.76) between the desire to use 
smartphones and tablets for laboratory and field work. This suggests that individuals who show a preference for using 
these devices for educational purposes are more likely to express a desire to use them specifically for practical activities. 

On the other hand, the correlation coefficient of 0.44, the lowest among the obtained results, means that par- 
ticipants are least likely to want to use smartphones and tablets to participate in distance education in biology classes. 
This indicates that respondents do not consider these devices to be particularly suitable or desirable for participation in 
distance education in biology classes. 

The results of the non-parametric Kruskal-Wallis test show statistically significant differences between the focus 
groups for six statements. However, the effect sizes between the groups are small or even negligible. Compared to par- 
ents, students and teachers showed somehow higher wishes to use smartphones for schoolwork, which was expected. 
However, biology teachers expressed a greater wish to use smartphones in biology classes than students and parents. 
This may suggest that biology teachers found the use of smart wearable technologies more meaningful and useful as 
students and parents. Nevertheless, it should be borne in mind that, according to the results in practice, the use of smart 
devices is only sporadic in the majority of cases. Therefore, further research should follow two tracks. Despite the signifi- 
cant growth and possibilities of mobile technologies, there is little empirical evidence on teachers’ perceived barriers 
to using mobile technologies in the classroom (Nikolopoulou et al., 2022), and basically no studies about their role as a 
supportive self-educational tool to establish learning methods. The use of mobile technologies could foster innovative 
pedagogical approaches to support/improve student learning; for this, the role, perspectives, and practices of teachers 
are very important (Nikolopoulou et al., 2022), however educational impact must be assessed when comparing with 
“traditional” school practices. Overall, the use of smartphones and tablets in biology classrooms can be a valuable tool 
to enhance the student learning experience and a tool to make lifelong learning an anytime-anywhere experience after 
formal education is over. 

Parents expressed the highest level of agreement with the wishes to use smartphones and tablets to participate 
in distance learning, especially for video conferencing. The finding most probably resulted from their involvement in 
remote education during the pandemic (Misirli & Ergulec, 2021). Nevertheless, they also expressed some reservations. 
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The small effect sizes suggest that these differences may not be significant, but they do suggest that further discussion 
and consideration are needed to address the concerns expressed by parents. 

When taken from a broader perspective, the knowledge about wishes must be enriched by studies and the prefer- 
ences and attitudes of these three groups because they may play an important role in shaping educational practice and 
policy. As students are the primary beneficiaries of technological advances, they can provide valuable insight into their 
comfort level, expectations, and preferences for integrating smartphones and tablets into their daily learning activities. 
Parents, in turn, have important views on the potential benefits, concerns and impact of technology on their children’s 
education. Finally, teachers, as facilitators of learning, have important knowledge and experience in integrating technol- 
ogy into their teaching strategies, and the good news of the study is that differences in wishes to use smartphones for 
educational purposes between all three groups are small or even negligible in terms of effect sizes. Open ended stays a 
discussion of a finding that mode and median of four (4 = occasionally) in all offered options except for online lectures, 
where numbers fall in median 5 (often) and mode 6 (very often), is optimal. Additionally unanswered and out of the scope 
of a recent study stays a question of whether the use of smartphones adds something positive to the development of 
lifelong learning skills, such as learning to learn, information, media, digital literacies, and problem-solving from a list of 
21% Century skills and competences. 

In summary, smartphones and tablets found their path in education. Therefore, it would not be a smart idea to 
stop them in front of the school doors. However, their use inside schools must be purposeful in order to teach students 
lifelong learning skills which is improbable to be learned through self-education. Maybe the way suggested by a teacher 
on one of the internet social network platforms can be by banning personal devices in classrooms and teaching what 
should be taught by the use of tablets in a controlled environment, and in such way reducing distractions and misuse 
of smart devices. And one of the ways can be to teach students how to use smartphones as data-collecting devices in 
biology laboratories and fieldwork. 


Conclusions and Implications 


The study analyses the intentions and wishes of students, teachers, and parents regarding the use of smartphones 
and tablets in biology lessons. The results indicate a generally positive attitude towards the use of these devices. The 
most desired use is for participation in distance learning, while other uses are categorized as occasional. The study finds 
that the use of sensors integrated into the devices during field and laboratory work is an interesting area for promotion. 
Only a minority of students expressed a desire not to use smartphones and tablets in the specified teaching scenarios. 
No teachers or parents fell into this category. Small but statistically significant differences were found between the 
focus groups, but students and teachers generally expressed greater agreement than parents. This is seen as positive 
as it suggests that the benefits and potential of smart devices in biology education could be better recognized. 

The suggestion is that school authorities should give teachers and students the opportunity to explore the 
potential of smart devices in a controlled environment. Direct bans are not recommended as they could do more 
harm than good. Overall, the study emphasizes the importance of evidence-based decision making and encourages 
further research into the potential benefits of integrating smartphones and tablets into the school environment. It 
recognizes the need for a balanced approach to avoid outright bans that could compromise educational opportunities. 
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Introduction 


High-level innovative talent is the key to national competitiveness. 
In the 1980s, the US National Science Board (National Science Board, NSB) 
proposed ‘STEM (Science, Technology, Engineering, Mathematics) educa- 
tion integration’ and developed it into a national strategy. In 2006, Profes- 
sor Georgette Yakman of the United States and his team added Arts as an 
essential humanistic factor to form the current concept of STEAM (Science, 
Technology, Engineering, Arts, and Mathematics) (Yakman, 2014). At present, 
STEAM education has been widely recognized by the education community 
(Connor, 2015). In recent years, STEAM education has received widespread 
attention from all over the world (Dejarnette, 2018; Tippett & Milford, 2017). 
Studies have shown that STEAM is a valuable resource for early childhood 
education and is very beneficial to young children (Clements & Sarama, 2016; 
Moomaw, 2016; Moomaw & Davis, 2010; Tippett & Milford, 2017). 

As the number of publications on early childhood STEAM (EC STEAM) 
education has increased dramatically, the research has evolved in its en- 
gagement with multiple subject matters, keeping in sync with the ensuing 
change and development. While such a change broadens the idea, on the 
other hand, keeping oneself up-to-date with evolution makes it a daunting 
and challenging experience. Therefore, it is demanding enough for research- 
ers and educators to comprehend the concept and to note its evolutionary 
trajectory simultaneously. This should hinder the progress that EC STEAM 
is expected to make in the near future. Currently, many review papers have 
been written from multiple vantage points, for instance, the use of STEAM, 
the impact of STEAM learning (Wahyuningsih et al. 2020), and Art and Cre- 
ativity Research (Perignat & Katz-Buonincontro, 2019). A comparative study 
of Thai and international research (2019) is also primarily comprehensive. 
The main research methods used are literature review (Wahyuningsih et al., 
2020), Perignat and Katz-Buonincontro (2019), the content analysis method 
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(Aysun & Hasibe, 2017), and the quantitative (coding system) and qualitative methods. Due to the limitations of 
the research method, a significant research gap can be found as existing literature on the subject is very limited. In 
addition, since the research on STEAM involves many disciplines, it is difficult to cover the entire research horizon 
and identify STEAM's research trend in one review article. Besides, there is a large degree of subjectivity in the selec- 
tion of literature for evaluation (Xie et al. 2020). To fill this gap, bibliometric analysis, first proposed by Pritchard in 
1969 (Fairthorne, 1969), uses mathematical and statistical methods to determine the state and future directions of 
a particular field of study. Thus, this research will emulate the objectivity and extent inherent in such methods to 
examine high-impact publications. The study performs cluster and network analysis, using Hiscite and CiteSpace 
applications to execute bibliometric investigation. In conclusion, the article also builds a knowledge map of EC 
STEAM education that facilitates readers’ objective and thorough understanding of the overall STEAM education 
scenario. The map helps understand the relationships between the multiple facets and aids in understanding the 
traits and trends of EC STEAM education. 


Focus and Objectives 


This study aims to conduct a scientometric analysis of academic publications in EC STEAM from 2008-2023, 
systematically studying various topics and related information in this field. More specifically, the main objectives 
of this research were as follows: 

1. Determine the research status, such as analysing the scientific growth pattern of publications, prolific 
journals, and citation structure of literature, among others. 

2. Identify key research topics and research results for influential articles. 

3. Analyse co-citation and co-occurrence networks to infer the knowledge landscape, research fronts and 
trends of EC STEAM. 

4. Build a knowledge map using the visual analysis of the literature as a foundation. 


Research Methodology 
Sample and Data 


Web of Science (WoS) provides researchers with online subscription-based indices (SCI, SSCI, SCIE, ESC), pro- 
viding categorization by year, subject, document type, publication type, source type, keyword occurrences, and 
country-by-country analysis of search results. This study uses the core data of the WoS database as the data source, 
with a total of 179 publications. They are systematically exported from the Web of Science database. 


Measures of Variables 


CiteSpace is a literature visualization tool that provides rich functions for researchers to identify knowledge 
links between publications, enabling users to analyse data by references, authors, sources, countries, and keywords. 
CiteSpace Version 6.2 was used for this study. Another analysis software used in this article was HistCite. It is a 
citation chronology visualization software developed jointly by Dr. Eugene Garfield and MG Science Publishing 
Company, which can graphically display the relationship between documents in a particular field, quickly map the 
development history of a specific field, locate important literature and the latest important literature in this field, as 
well as essential scientists and institutions. The author used HistCite to display the development history of the EC 
STEAM field graphically, used CiteSpace co-occurrence and co-citation map analysis to display the development 
trend of EC STEAM, and summarized and sorted out the research direction and hotspots of EC STEAM based on 
an in-depth reading of classic literature. Figure 1 represents the main analysis items and tools used in this study. 
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Figure 1 
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Note. *E represents Excel; H represents HistCite; C represents CiteSpace. 
Note: The consideration period for evaluated literature is 15 years (i.e., 2008-2023), with one year being selected as a time slice 
for analysis. 


Data Analysis Procedure with Search Strategy 


This review focuses on the papers on “STEM for young children” and “STEAM for young children.’ At the same 
time, the search strategy of (citation topics meso) “education & education research” was selected, and a total of 
183 articles were collected on April 2, 2023. To ensure the accuracy of the data, the researcher reviewed the titles 
and abstracts of all articles separately, and 156 articles met the requirements of the collected data. Then, through 
HistCite’s document co-citation analysis function, 23 highly cited articles were added, so a total of 179 publica- 
tions were collected for the analysis. The data were exported in “plain text” by selecting the “Full Record and Cite 
References” option for subsequent analysis by CiteSpace. 


Research Results 
Literature Analysis 
Overall Trend Analysis of Published Papers 


This section deals with the trend in the number of studies on EC STEAM. The trend line was drawn with the 
help of Microsoft Excel (Figure 2), charting the number of publications in the corresponding year between 2008 
and 2016. It shows a general upward trend since 2008, with a major slope break following the year 2016. However, 
the publication took a downward spiral between 2018 and 2019, only to take to a newer height between 2021 and 
2022. Such an unprecedented increase took place despite the global pandemic when accessing physical research 
infrastructure was nearly impossible. This surge, however, could be attributed to two factors: either the research 
community's zeal to conduct adequate research even during episodes of global restriction or utilizing that time to 
prepare a robust research blueprint for pursuing it as the global restrictions were gradually withdrawn. At the outset, 
this change in the number of publications corresponds to the development trend of STEM education research. All 
the United States Presidents since the 1990s have taken a keen interest in the propagation and the development 
of STEAM education; they have also appealed for the achievement of national competitiveness through STEAM 
education. In keeping hooked to the presidential aspirations, the government officially passed the “STEAM Educa- 
tion Act of 2015” on October 7, 2015. These developments reaffirm the critical position of STEAM education. At the 
end of 2018, a five-year plan strategy was established for the successful implementation of STEAM education at 
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the international level. The United States devoted its efforts to the federal government, combined with industry, 
government, academia, and Congress, to announce the implementation of STEAM education, which is regarded 
as a development that is critical to the country's competitiveness. Such initiatives found global stimulus and con- 
tributed to a substantial increase in research articles on STEAM education in 2015 and 2016. 


Figure 2 
Trends of EC STEAM papers based on the Web of Science during 2008-2023 


Number of studies 


2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 


== Number of studies 


Document co-citation visual analysis 


The top 30 documents (based on citation count) were collected and sorted by LCS (local citation score). A cita- 
tion chronology map was prepared (Figure 3) with the help of the visual analysis tool - HistCite. As shown in Figure 3, 
the most relevant documents published between 2008 and 2023 are frequently cited and form a close relationship 
with each other. The author of the three most cited articles is Jamil, whose work on teachers was published in 2018. 
It was followed by an article written on Empirical Research on Robotic Activities by Kazakoff and was published in 
2018. The third highly cited paper was written by Park in 2017, which engages with teacher's beliefs. The research 
has grown even more significantly in 2017-2018. The graph illustrates how the number of publications increased 
dramatically in 2017. This year coincided with the growing interest in teachers’ beliefs within the larger concerns 
of EC STEAM. The earliest node in this regard, as seen in Figure 3, is Article 10. This article by Brophy et al. (2008) 
employed an analytical frame to study P-12 engineering education. To have an in-depth analysis, these articles 
were further classified to chalk out the thematic ideas inherent in them (Table 1 Appendix). 

The thematic categorization of 30 highly cited articles hints at the prevailing current research on EC STEAM 
education that has a relatively great influence and also underlines the future development trend. The following 
seven thematic categories are extracted: 

1) ECSTEAM subject research: STEAM education is a comprehensive form of education that includes sci- 
ence, technology, engineering, arts, and mathematics. In early childhood, each of these subjects adds 
to the growth of enquiry among children. To substantiate that, a survey (Maltese, 2010) shows that early 
scientific interest is a significant endeavour for development. Engineering is the foundation of STEM, and 
some studies have noted that engineering education can be carried out in early childhood education 
and is very important in contemporary times (Bagiati & Evangelou, 2015; Brophy et al., 2008). Young 
children’s innate curiosity and desire to inquire and explore the world not only form the cornerstone 
of early childhood development (Brophy et al. 2008) but also allow them to be creative, innovative 
and improvisers, much like engineers (Tippett and Milford, 2017). In technology and computational 
thinking, the primary learning centres have been around robotics since early childhood (Sullivan & Bers, 
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2016; Bers et al., 2014; Sullivan & Bers, 2018; Kazakoff, 2013). Research by Master et al. (2017) notes that 
incorporating non-academic social factors (such as group membership) into current STEM curricula 
may be an effective way to increase STEM motivation in young children. The five disciplines in early 
childhood STEAM education are interrelated and mutually reinforcing to constitute the integral core 
of STEAM education. In addition to the above disciplines, STEAM education often incorporates other 
disciplines as a supplement. For example, Strawhacker and others (2020) carried out bioengineering- 
related STEAM activities, which they certainly argue have the potential to engage young learners in 
the context of real-world challenges in scientific inquiry and engineering design. 

Research on EC STEAM teachers: In STEAM education, teachers play a vital role. Jamil (2018) identifies 
early childhood teachers as primary adopters and implementers of changing educational models. 
Their belief in new approaches is an invaluable resource guiding educational innovation, whose value 
gets amplified when practised in classrooms. At the same time, preschool teachers also need to have 
experience in implementing STEAM education, for which they need to be trained in their knowledge, 
ability, and teaching methods (Alghamdi, 2023). Training affects beliefs in the implementation of STEAM 
education for in-service preschool teachers and preservice teachers. (Alghamdi, 2023). It also signifi- 
cantly improves their knowledge pool (Park, 2017) and ability to teach (Chen et al., 2021). Despite the 
increasing emphasis on teacher training, teachers still encounter various problems in carrying out STEAM 
activities, as they face practical constraints of time, family and peer support, resources, differences in 
each student's ability to grasp, and safety concerns (Wan et al., 2021). However, more concerning is the 
teachers’ lack of professional knowledge and skills in various STEAM disciplines. It makes it difficult for 
them to plan courses based on STEM education (Yildirim, 2021) and to carry out integrated education 
based on real-life relevancy (Alghamdi, 2023). 

EC STEAM curriculum research: In addition to teacher training, curriculum development is also a good 
way to promote the development and promotion of STEAM education. It is one of the crucial recourses 
to several difficulties and problems faced in the development and implementation of STEAM educa- 
tion. Scholarly research has substantiated this concern with suggestions for developing curriculum 
frameworks. For example, Dubosarsky et al. (2018) explain the research conducted by the Seeds of 
STEM team to build a curriculum for children belonging to culturally and linguistically diverse back- 
grounds. It further explains the modes inherent in developing curriculum frameworks, the perfection 
of its implementation, and its ultimate effectiveness. Brenneman et al. (2019) devised a professional 
development model that enables preschool educators to deliver rich and high-quality STEM learning 
experiences. 

EC STEAM parent research: In addition to retrofitting teacher training and course moderation, parent's 
support for pursuing STEM is equally important. Numerous studies (Tay et al., 2018; Tippett & Milford, 
2017) indicate that young children and their parents have fairly positive attitudes toward STEM learning. 
In addition to educational value, the economic value of STEM learning to their families is also recognized. 
Parents believe that learning STEM can pave their children’s path to future economic prosperity. At the 
same time, parents also need more training on how to work with their children in STEM activities and 
build support for their children’s learning (Wan et al., 2021). 

EC STEAM in different countries: Although STEAM education originated in the United States, it has set 
off a wave in education all over the world. Research on STEAM in different countries helps to understand 
its application and effects in different cultural contexts, such as Australia (Simoncini & Lasen, 2018). 
Informal STEAM activities: In addition to formal STEAM education, informal STEAM education also acts 
as an important cornerstone (Marcus, 2018). Investigation of ways to support young children’s STEM 
learning and knowledge transfer through informal learning experiences, as in museums, suggests its 
far-reaching implications. Thus, providing impetus for establishing informal practices as an important 
tool for the propagation of STEM education. 

Gender difference: Gender differential also forms one of the key subject matters of research as STEM 
education is mostly science-based and technical. Master et al. (2017) pointed out the existence of a 
large and persistent gender gap in science, technology, engineering, and mathematics. They underlined 
activities that ought to provide more opportunities for young girls to experience technical activities, 
which consequently will increase interest in STEAM education. 
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Figure 3 
Citation network of EC STEAM research on the Web of Science 


Keyword Analysis 


Note: *Year (number of studies) is presented in the left column. Each circle represents an article, and the number in the box is the 
article's ID. 


Analysis of EC STEAM Keyword Co-Occurrence 


Keyword co-occurrence analysis is an effective way to clarify scientific knowledge structure and hotspots and 
discover research trends. This article has employed the CiteSpace application to generate a keyword co-occurrence 
graph (Figure 4); it depicts the keyword map of EC STEAM research from 2008 to 2023, in which a total of 211 
high-frequency words and 829 connecting lines are charted. Normatively, the higher the frequency of keyword 
citations, the more the research content is valued. The betweenness centrality measures the frequency with which 
a node in the knowledge graph acts as an intermediary for other nodes to connect. Keywords with a central value 
exceeding 0.1 (see Table 2) have a greater influence (Fang et al., 2017). In co-occurrence analysis, nodes with high 
frequency and high concentration are usually considered key nodes, indicating that they have a strong influence 
on the whole network. By studying these key nodes, the rich information behind the nodes can be mined. Since 
EC STEAM started in 2008, science and design have been relatively active keywords, which underlines the fact that 
research on EC STEAM often focuses on scientific inquiry applications and engineering design. Since 2017, some 
keywords, such as gender, knowledge, achievement, educational robotics, identity, experience, and curriculum, 
have appeared more frequently, indicating that studies have begun to focus on gender differences, STEAM educa- 
tion emotional identity, experiences, and knowledge gain. In addition, curriculum research has gradually begun, 
with educational robots as relatively more often used technologies and matter of educational content. However, 
the centrality is low on these topics, which also means that no new hotspots have been formed in these research 
areas. The reason for the absence of hot spots is the limited promotion and development of EC STEAM practice 
and research during this period. Besides, the practical trials and research by teachers and researchers remained 
small ona scale. 
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Figure 4 
Keyword Co-Occurrence Network of EC STEAM 
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Keywords of High Frequency and High Betweenness in EC STEAM during 2008-2023 


No. Keyword Year 
1 Science 2008 
2 Gender 2017 
3 Education 2017 
4 Knowledge 2017 
5 Early Childhood 2014 
6 Achievement 2017 
7 Design 2008 
8 Students 2008 


Keyword Cluster Analysis 


Betweenness Frequency 
0.32 24 
0.16 5 
0.14 13 
0.14 7 
0.13 17 
0.12 6 
0.111 6 
0.10 5 


The CiteSpace clustering function is employed to extract information using keywords. The logarithmic 
likelihood ratio (LLR) is used as a calculation method to obtain clustering results, which produces eight effective 
clustering labels (see Figure 5). Among them, the silhouette score above 0.5 indicates that the clustering result is 
reasonable, and the modularity Q value of 0.5885 (above the threshold of 0.3) indicates that the clustering structure 
is reasonable. For the convenience of discussion, this article puts #0 (i.e., early childhood) and #3 (i.e., early child- 
hood education) together to reduce the redundancy error. Cluster analysis and discussion of keywords can help 
teachers and researchers systematically and deeply understand the research content of EC STEAM. 
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Figure 5 

Keyword Cluster Analysis 
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Children between the ages of three and six are considered to be in their early childhood. There is a widespread 
belief that STEAM education is too advanced for young students. This idea appears to be a common misconcep- 
tion. It is completely at odds with the wealth of academic studies that highlight children’s innate curiosity and 
aptitude (for instance, see Trundle & Sackes, 2015). In their study, Tippett and Milford (2017) collected data through 
classroom observations and surveys with multiple stakeholders (teachers, students, and parents) to examine how 
STEM activities were integrated into Pre-K. The findings support the inclusion of STEM-based learning in young 
children. In early childhood education, games (Hollenstein, 2022), stories (Emmons, 2018), and robots (discussed 
later) are fundamental modes through which young children are made more interested in STEM. For EC education, 
STEAM activities can promote the development of children’s knowledge, ability and thinking in all aspects, includ- 
ing mathematics, as mentioned above (Hollenstein, 2022), computational thinking, robotics, programming, and 
problem-solving abilities (Bers et al., 2014), scientific knowledge (Emmons, 2018), and artistic and social-emotional 
aspects (Garner, 2018). STEAM education also helps to develop young children’s scientific inquiry ability (Schiefer, 
2017; Brophy, 2008) and can also promote young children’s collaborative problem-solving (CPS) behaviour (Herro, 
2021). Due to the age-specific nature of EC, STEAM education also relies heavily on the guidance of teachers and the 
support of parents. Early childhood education professionals play an important role in supporting young children’s 
interest and participation in STEM education (Simoncini & Lasen, 2018). Teachers need professional development 
on how to integrate STEM into the preschool curriculum and how to design experiences that support classroom 
learners to provide them with a high-quality education. Brenneman’s (2019) research designed a teaching model 
for teachers to carry out STEAM activities better. In early childhood education, parents play an important role in 
family education. Tay (2018) pointed out that parents have a positive attitude toward STEAM education. 

1) STEM Education 

STEM (Science-Technology-Engineering-Mathematics) education has received significant attention in recent 

years. This has not only increased interest and learning in these fields but also encouraged children and young 
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people to pursue careers in these fields. Early exposure to STEM innovations and learning are important predictors 

of children’s future participation in STEM careers (Correia, 2022). STEAM education is proposed to cultivate com- 

prehensive talent for future development. Studies have shown that STEAM education can have a positive impact 

on the job choices of various groups, such as girls (Emembolu, 2020) and students in rural areas (Brownlee, 2021). 

Carrying out STEAM research at the outset can effectively influence learners’ future career choices at an early stage. 
2) Attitude 

In STEAM education, the knowledge and skills related to the subject are essential, and there is a certain de- 
gree of complexity in its comprehension. This implicates a need for persistent motivation, interest, and learning 
enthusiasm to remain in the field. Maltese (2010) demonstrates that interest in science should begin in early child- 
hood by examining scientists’and graduate students’ first experiences in science. Master et al. (2017) pointed out 
that improving children’s enthusiasm for STEM from an early age is an effective way to improve children’s STEM 
motivation, and it can also improve children’s self-efficacy. 

4) Natural Education 

Some aspect of STEAM education also draws its relevance from natural and atmospheric sciences, such as 
learning about Climate Change (Trott, 2020) — the author has investigated how climate change learning and ac- 
tions affect children’s cognitive, emotional, and behavioural engagement with science. Between 2015 and 2020, 
Speldewinde and Campbell (2021) implemented STEM education in Jungle Kindergarten to find out that nature, 
being an essential element of STEAM education, is also a significant factor in cultivating children’s interest in science. 

6) Cultural Adaptive Pedagogies 

Studies on STEAM education often consider the importance of sociocultural background, thereby insisting 
on adding S (social) components to STE (science-technology-environment) education (Zoller, 2011). Smith (2022), 
while pursuing a critical analysis of the state of science education in the Netherlands, provides a concrete example 
of a culturally relevant and sustaining pedagogies (CR-SP) and community-based STEAM program for young chil- 
dren and their parents in the north of the Netherlands. It counters the prevalent European hegemonic argument 
by locating Dutch society as the mixing pot of European and Caribbean cultures. 

7) Gender Differences 

In STEAM education, because science, technology, and engineering are all scientific and technical disciplines, 
there are often gender differences. Conlon’s (2023) survey found that boys are more likely to engage in military, 
manual labour, and mathematics/computer science careers, while girls are more likely to be caregivers in stay-at- 
home parenting, education, and animal care. Master et al’s (2017) survey shows a significant and persistent gender 
gap between boys and girls in Science, Technology, Engineering and Mathematics (STEM) Participation; this gap is 
much more significant in technical fields such as computer science and engineering than in mathematics and science. 

8) Robotics for Early Childhood Education 

Educational robots are a relatively important component of STEAM, especially for young children. In robotics 
education, we find different types of concepts, methods and goals surrounding robotic technologies in addition to 
several educational robots. Sullivan (2016) noted that robotics provides a fun and tangible way for children to get 
exposed to technology and engineering experiences during the foundational stages of early childhood. Robotics 
education also provides learners with hands-on experience in understanding technology, machine language, and 
systems (Eguchi, 2014). Research (Ferrada-Ferrada, 2020) has shown that starting in preschool, children are able to 
acquire basic robotics and programming skills. 

In summary, as the visual analysis of literature cross-citation consumes time, the cluster analysis of keywords 
is more economical in terms of time. From the cluster analysis, we can see that carrying out STEAM education in 
the early childhood stage can affect the future employment direction of young children (especially girls) and can 
also stimulate children’s interest, curiosity, and self-efficacy. It also helps them in their careers and allows them to 
keep learning and improving continuously. STEAM education is of great value to the overall development of young 
children, and we should study and practice it from all aspects. 


Keyword Citation Burst Analysis 


Business reflects the increase in the frequency of document citations in a certain period. Through the detec- 
tion of burstiness (Fig. 6), we can determine the research hotspots during that period. Using the CiteSpace burst 
detection algorithm, it can be seen that the research frontier of EC STEAM is divided into three stages: 

1) 2008-2015 is a stage of relatively slow development, and the research hotspots are mainly focused on 
students, performance, and representations. 
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2) Since 2017, research has begun to emphasize science education and technology subjects and takes a 
keen interest in the classroom, curriculum, and gender. In addition, it emphasizes elementary educa- 
tion, elementary students, and early childhood education. It is during this moment that it begins to 
pay attention to the significant role of teachers — instruction and beliefs. 

3) From 2020 to the present, there has been a new research trend which takes keen note of children’s 
experience and requires professional development in STEAM. 

This emphasizes the experiential nature of EC STEAM activities and the requirements for activities. The de- 
velopment of EC STEAM education has shifted from emphasizing the teaching of STEAM knowledge and skills to 
emphasizing young children’s inquiry and experience. It demonstrates EC STEAM education's increasing ability to 
pay attention to learner characteristics, which is related to their physical and mental development. Thus, it prioritizes 
better influence of STEM that accentuates children’s faculty development. 


Figure 6 
Top 16 Keywords in EC STEAM with the Most Robust Citation Burst 


Top 16 Keywords with the Strongest Citation Bursts 


Keywords Year Strength Begin End 2008-2023 

students 2008 0.84 2008 £2011 

performance 2010 1.24 2010 2011 

representations 2011 1.19 2011 2015 a 
science education 2010 1.27 2017 2018 RR 
classroom 2018 1.94 2018 2019 — 
instruction 2018 0.96 2018 2019 ———— 
curriculum 2018 0.81 2018 2019 = #8327 
elementary education 2014 0.75 2018 2019 —— 
beliefs 2019 1.41 2019 2020 = 
gender 2017 0.96 2019 2020 ———E 
stem education 2020 3.55 2020 2021 ———— 
technology 2008 1.32 2020 202) —___ =! 
elementary students 2020 0.85 2020 2021 — 
experiences 2020 0.82 2020 2023 —<—= 
early childhood education 2017 0.3 2020 2021 LL 
professional development 2019 0.64 2021 2023 —_—— 

Author Analysis 


320 Authors in HistCite 


Using HistCite software analysis (Table 3, see Appendix), it was found that there are 179 articles in the current 
database, four of which are written by the author Marina Umaschi Bers. The total citations for her research are 245, 
which signifies a relatively high-yielding and influential author. In addition, the author Ceylan has their articles 
and is also a relatively productive author. Authors Cheryan, Master, Meltzoff, Sullivan, Brophy, Klein, Moscatelli, 
Portsmore, and Rogers have relatively high local citation scores and international citation scores and are relatively 
influential authors. In Table 3, there are links to the introductions of highly productive authors, mainly including 
names, work units, research fields and published articles. 
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Coauthor network 


Using CiteSpace, the coauthor network diagrams can be generated. Coauthor networks are used to analyse 
joint research in a specific research field. The larger the nodes in the graph, the more publications are published, 
and the thicker the connecting lines, the closer and more robust the collaboration between authors. As shown in 
Figure 7, the CiteSpace analysis results show that the rate of collaboration among authors is meagre. A relatively 
stable and sizeable academic team has yet to be formed. 


Figure 7 
Author Co-Occurrence Analysis 
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Author Co-Citation Analysis 


Author co-citation maps (Fig. 8) are generated using CiteSpace software. The analysis of author co-citations 
yielded a total of 321 nodes and 892 network lines, as shown in Figure 8. The larger the node in the graph, the 
more frequently it is referenced. Clements (14), Bers (13), Sullivan (12), and Eshach et al. (10) have the most cited 
and strongly centred results and show that they have played a significant role in STEAM education research. 
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Figure 8 
Author Co-Citation Analysis 
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Journal Analysis 


Using HistCite software analysis, a total of 79 journals were used, among which the “Early Childhood Education 
Journal” published the most articles (7) and consequently received relatively more citations.“European Early Child- 
hood Education Research Journal” and “International Journal of Technology and Design Education” (3) published 
more articles. At the same time, “The Journal of Engineering Education” had only one article, but its citations were 
from more significant and critical journals. 


Institutional analysis 
Number of institutions 


Among the large number of institutions that have contributed to research in STEAM education research, Tufts 
University, Queensland University, Purdue University, and Texas A&M University have published more than three 
articles each. Their citation scores are relatively high, thereby becoming high-yielding and influential institutions. 
However, the University of Washington, despite publishing two articles, shows a considerably high citation score. 


Joint research institution network 


CiteSpace analysis (Figure 9) shows that there are 119 research institutions and 126 cooperation links in the 
joint research institution network. Figure 10 shows that most of the nodes are isolated points, indicating that 
almost all research results were done by a single author. Only a few organizations have conducted collaborative 
research, signifying a weak intensity of collaboration. In terms of the number of publications, the top three are Tufts 
University (7), Queensland University of Technology (QUT) (5), and Texas A&M University System (5). In this study, 
the maximum number of publications by any institute is only seven, which is a relatively small number of publica- 
tions. This indicates that the research on EC STEAM by these institutes is not extensive vis-a-vis the global concern. 
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Figure 9 
Co-occurrences of the Institutions 
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Analysis of National Literature Citation Rate 


The scholars in the USA have published the most papers in the field of EC STEAM - 47 papers. Their citation 
score is also the highest (1124), signifying their notable influence in this field. Australia (14), Turkey (13), the UK 
(9), and the People’s Republic of China (5) are distantly ranked in comparison to the USA but have recognizable 
citation strengths. Although South Korea has only one article, it has a very high citation score, hinting at some 
influence it commands. 


National (regional) network 


CiteSpace was used to draw a national cooperation network map. The size of the node (Figure 10) represents 
the degree of cooperation. The larger the node, the more the country’s cooperation with other countries. It shows 
that there are a total of 25 countries and regions and nine cooperation links. The countries with the highest num- 
ber of items are the United States (46), Australia (12), Turkey (7), United Kingdom (4), Spain (4), Canada (3), China 
(3), and Greece (3). Using node concentration as a metric, it can be seen that the United States, Australia, and the 
United Kingdom have played critical roles in this area of research. Betweenness centrality represents a wide range 
of national cooperation objects at essential nodes in the international cooperation network. The higher the be- 
tweenness centrality is ranked, the more significant the contribution to the formation of the network pattern and 
structural stability. Table X shows that the USA has the highest centrality (0.09), followed by Australia and the UK 
(0.04), and the others are all 0.00, indicating that there are few transnational (regional) links during the research 
process. While links between countries and regions often exist, as shown by the collaborative lines in the network, 
the strength of the links is weak. It suggests that the endeavour to establish such linkages across countries and 
regions still needs to be more stable. 
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Figure 10 
Co-occurrences of Countries/Regions 


e gone, 
é (46) USA, 
° ws 
APAN 


Analysis 


The examination of research patterns relating to Early Childhood STEAM education reveals interesting facets 
across the years. The available evidence indicates a notable increase in the quantity of research conducted on this 
specific topic starting in 2008. The observed pattern indicates an increasing preference for probing the possibilities 
of including STEAM principles in early childhood education. The data also finds noteworthy fluctuations in research 
intensity, wherein there are visible increases and decreases that align with global events, such as the COVID-19 
pandemic. This phenomenon exemplifies the tenacity and flexibility of the academic community, which persisted 
in the face of difficult situations. 

Keywords have a crucial role in comprehending the thematic landscape of research on early childhood STEAM 
education. The research shed light on the significant terms, their patterns of co-occurrence, and the formation of 
these clusters. The analysis highlights the changing emphasis of research over some time. Between 2008 and 2015, 
the research primarily focused on fundamental elements, including students, performance, and representations. 
This initial study possibly established the basis for further inquiries. 

One notable finding relates to the transition towards a broader emphasis on several dimensions of education, 
encompassing science education, technology-related courses, classroom dynamics, curriculum creation, and gender 
disparities. This evolutionary process is in accordance with the broader pedagogical transition towards compre- 
hensive and inclusive education. Significantly, there has been a visible movement in focus within the educational 
landscape towards topics such as experiential learning, emotional identity, and professional development. This 
shift talks about the departure from the customary emphasis on the acquisition of knowledge and skills to a move 
towards the cultivation of inquiry, curiosity, and the efficacy of students and instructors alike. 

The concept of Citation Burst Analysis (CBA) refers to a methodological approach used in academic research 
to analyse the temporal patterns of citations within scholarly literature. The examination of citation bursts offers a 
noteworthy understanding of the temporal dynamics of research key points. The boundary of three discrete phases 
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signifies the development and advancement of the Early Childhood STEAM research domain. The early stages of 
this phenomenon were primarily focused on conducting fundamental research. However, as time progressed, 
there was a shift towards prioritizing pedagogical methods, using technology, and gaining a more comprehensive 
understanding of the learner’s experiences. The aforementioned temporal study underscores the mutability inher- 
ent in the research landscape. The subjects experience fluctuations in both prominence and impact as a result of 
altering educational paradigms and societal demands. 

The analysis of authors, journals, and institutions enhances comprehension of the critical scholars who are 
crucial in shaping research in the field of Early Childhood STEAM education. The study highlights the scholarly 
contributions of notable and essential academics like Marina Umaschi Bers and Remziye Ceylan. The contributions 
made by individuals in this interdisciplinary subject are indicative of the wide range of skills that are necessary. 
The analysis of the coauthor network unveils visible patterns of collaboration, highlighting instances of both 
unaccompanied and restricted joint endeavours. This implies potential avenues for cultivating broader research 
partnerships to further the communication of knowledge and encourage innovation. 

The examination of journals and institutions sheds insight into the key channels and centres of research relating 
to Early Childhood STEAM. Prominent scholarly journals such as “The Early Childhood Education Journal;”European 
Early Childhood Education Research Journal,” and “International Journal of Technology and Design Education” 
play a crucial role in the dissemination of research findings. The analysis highlights the significant contribution of 
institutions such as Tufts University, Queensland University, and Texas A&M University in generating research with 
substantial influence. The extent of collaborative networks among institutions appears to be constrained, indicating 
the potential for expanding collaborations in order to improve the quality and extent of research. 

The analysis of national and regional contributions to research on Early Childhood STEAM education finds 
interesting trends of influence and collaboration. The United States’ pre-eminence in publishing and citation score 
highlights its position as a frontrunner. Nevertheless, the extent of collaborations among countries and regions still 
needs to be expanded, suggesting the possibility for enhanced international partnerships. The acknowledgement 
of additional significant contributions, namely from Australia, Turkey, the United Kingdom, and China, highlights 
the international scope of research in early childhood STEAM education. 

The study provides a thorough examination of Early Childhood STEAM education research, focusing on its 
development, prominent topics, notable participants, and worldwide trends. The patterns indicate a transition 
towards comprehensive approaches that incorporate experiential learning, gender issues, teacher responsibilities, 
and pedagogical innovation. As the discipline progresses, a number of implications arise: 

The pedagogical transformation refers to the shift from a traditional knowledge-centred approach to a more 
experiential learning model, which is in line with educational paradigms that encourage active participation and 
inquiry. The transition above has significant consequences for the preparation of teachers, the development of 
educational programs, and the allocation of resources. 

Gender equity refers to the acknowledgement of distinctions between genders and the undertakings made 
to rectify any inequalities. It demonstrates the academic dedication to foster an all-inclusive educational milieu. 
Further investigation is necessary to explore more extensively the practical approaches for nurturing equality in 
STEAM disciplines. 

The current lack of extensive collaboration among authors, institutions, and nations indicates the potential for 
cultivating interdisciplinary and transnational research relationships. Collaborative endeavours have the potential 
to advance the dissemination of knowledge and bring innovation from multiple vantage points. 

The incorporation of technological courses and educational robots highlights the significance of technology 
in augmenting early childhood education. Conducting an inquiry into the effectiveness and influence of these 
technologies on educational achievements might yield significant scholarly perspectives. 

The implications for policy are evident in the examination of publication patterns and the impact of policy 
measures, such as the “STEAM Education Act of 2015,’ indicating a correlation between educational policy and 
research inclinations. Further investigation is necessary to delve into this connection in order to provide valuable 
insights for policy-making. 

In conclusion, the examination of research patterns and dynamics in Early Childhood STEAM education 
highlights the progressive nature of the discipline, its multidisciplinary nature, and its capacity to influence the 
educational domain. The analysis yields valuable insights that can serve as a basis for making informed decisions, 
identifying future research avenues, and fostering collaborative endeavours to improve the quality of early child- 
hood education and equip young learners for a future characterized by dynamism and innovation. 
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Discussion 


The study on Early Childhood STEAM education presents a wide-ranging scientometric analysis of research 
conducted between 2008 and 2023. The study's employing the Web of Science database ensures a comprehensive 
and structured approach to data collection while acknowledging the potential limitations of any single database. 
It includes, inter alia, restricted coverage of non-English journals or journals that might offer unique perspectives 
on the subject matter. Besides, while CiteSpace and HistCite are robust visualization tools, their selection might 
introduce certain biases or limitations in capturing the entire research landscape. 

The observed surge in research activity from 2008 aligns with global trends toward incorporating STEAM into 
early childhood education. However, the study might benefit from a more intense exploration of the impact of 
specific global events like COVID-19 pandemic, on research focus. Understanding how such events shaped research 
trajectories may offer valuable insights into the field’s status. The evolution of keyword clusters over time offers 
a compelling view of the changing emphasis of research. However, the study could investigate the contextual 
contingencies behind these shifts. For instance, by exploring the role of societal changes or educational policies 
as they influence the evolution of research topics. 

While acknowledging influential scholars and key institutions is commendable, a deeper exploration into 
their specific contributions and methodologies could provide richer insights. Understanding the methodologies, 
theories, and approaches that these scholars and institutions have employed could highlight crucial trends or 
innovative practices in Early Childhood STEAM education. Moreover, while identifying patterns of collaboration 
among authors and institutions, the study further explores the quality and impact of these collaborations. Inves- 
tigating the nature of collaborative research and its contributions to the field could offer valuable insights into the 
effectiveness of interdisciplinary studies. 

The study's emphasis on national and regional contributions is insightful which highlights dominant countries 
and emerging contributors. However, the analysis could benefit from a more detailed examination of cultural, so- 
cietal, or policy factors that influence research priorities and approaches in different regions. This could provide a 
deeper understanding of the global contextual influences on Early Childhood STEAM education. Exploring interdis- 
ciplinary intersections beyond the realms of science, technology, engineering, arts, and mathematics (STEAM) within 
Early Childhood Education could uncover new dimensions and perspectives essential for holistic development. 

The present study provides valuable insights into the dynamic nature of Early Childhood STEAM education 
research. In addition, effectively exploring contextual influences and overcoming potential limitations could further 
enrich and deepen our understanding of this multidisciplinary field. Collaboration among researchers, interdisci- 
plinary approaches, and a nuanced understanding of global contexts will be instrumental in shaping the future 
trajectory of research in Early Childhood STEAM education. 


Conclusion and Suggestions 


This study conducted a visual bibliometric analysis of the literature by employing Hiscite and CiteSpace ap- 
plications. Then, it methodically analysed and interpreted the research content and information, such as authors, 
institutions, journals, and countries of origin. This article used a diagram to help explain the direction of research 
on STEAM education. In conclusion, the EC STEAM knowledge map (See Appendix) is an organic flow diagram that 
ties together the elements necessary to deliver effective STEAM education. 

The number of scholarly pieces collated and studied in this essay is on the higher side vis-a-vis the existing 
literature. Most importantly, the selection of literature is objective, and the information extracted from it is com- 
prehensive and exhaustive. 

The observed trends highlight how the field has evolved from knowledge to a comprehensive approach that 
encourages experiential learning, involves teachers, and promotes gender equality. This transformation supports 
thinking and the broader societal need for innovative and well-rounded education. 

The emergence of clusters, keywords and patterns of activity reflects how this field responds to evolving edu- 
cational paradigms and global events. The contributions made by authors, institutions and countries emphasize 
the nature of research while also presenting opportunities for interdisciplinary cooperation on a broader scale. 

Several implications can be drawn from this analysis. Firstly, the necessity for transformation, integration of 
technology into childhood education and an approach that prioritizes equity. Secondly, the impact of efforts in 
driving innovation and strengthening research outcomes. Lastly, it underscores the relationship between academia 
and educational administration by highlighting how policy initiatives influence research trends. 
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As Early Childhood STEAM education continues to evolve, this analysis serves as a guiding reference for re- 
searchers, educators, policymakers, and institutions. It emphasizes the importance of adopting an approach that 
nurtures a learner’s natural curiosity while developing critical thinking skills, creativity, and problem-solving abilities. 

These insights are potent enough for developing a dynamic, inclusive, and impactful early childhood STEAM 
education that empowers the upcoming generation to thrive in a world that is constantly evolving. 

The study takes the initiative to provide some suggestions. Firstly, the current state of STEAM education for 
young children is dynamic and multifaceted, with constant interaction of various facts of learning and evolution. 
We can only continue to support the growth of early childhood STEAM education by synthesizing the content 
and importance of all aspects in understanding and conducting STEAM education. Secondly, in keeping with 
the global requirement, the government should provide support for policies and funds. Thirdly, there must be 
a system of resource pooling for a shareable knowledge environment to prosper. Children’s STEAM education 
will attain more impact if high-quality educational resources are disseminated universally through the Internet. 
Fourthly, employing more diversified education methods will have a positive outcome. Making course contents 
colourful, employing virtual and augmented reality, and emphasizing practical experiments and field exploration 
can make STEAM education more acceptable to curious young minds. Fifthly, the student evaluation must not only 
be based on grades but must also focus on evaluating their various abilities - practical aptitude, innovation ability, 
and teamwork skills. Sixthly, keen interest ought to be taken in training teachers who impart STEAM education to 
children. Lastly and most importantly, the economic, social, and global relevance of STEAM education for children 
needs to be presented to the parents. They are the ones who determine the career trajectory of their children by 
opting for schools, paying for their education, and creating a learning environment at home. The prescriptions 
mentioned above can only be materialized with the parent's consent. 

Research work, being a benchmark, dictates covered distance and yet-to-be-covered distance at once. To 
further the cause of global outreach, more collaborative research endeavours are required. Collaborations enable 
a transactional space where learning from each other's experiences seeps through the policy prescriptions for a 
larger good. 
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Abstract. Biotechnology has great 
importance as a socioscientific issue in 
STEM education. This study aimed to 
explore the knowledge and attitudes of 
college students toward biotechnology. A 
total of 236 university students participated 
in the study. Questionnaires on knowledge 
and attitudes towards biotechnology 

were used for data collection. The results 
revealed that participants had inadequate 
knowledge about several basic concepts 
related to genetics and biotechnology, 
particularly those concerning genetically 
modified organisms. The results regarding 
the attitudes revealed more neutral results. 
The participants had a neutral attitude 
towards biotechnology, with mean scores 
ranging from 2.71 to 3.55. They were 
generally against buying genetically 
modified products but supported 
biotechnology for medical purposes 

and strongly desired to increase their 
knowledge about genetically modified 
products. The participants did not have 

a negative opinion of biotechnology in 
general but were critical of its purpose 
and use. The correlation analysis between 
knowledge and attitudes produced weak 
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Introduction 


The conjunction of science, technology, engineering, and mathemat- 
ics (STEM) has encouraged innovative developments, exerting a profound 
influence on society. Biotechnology is central to this interdisciplinary field, 
which connects the power of living organisms, cells, and biological systems to 
revolutionize industry, medicine, and agriculture (Erdogan et al., 2012; Singh 
et al., 2019). With its transformative potential, biotechnology represents the 
convergence of biology, chemistry, and technology and provides a dynamic 
platform for students to engage with real-world applications of scientific 
principles (L6pez-Banet et al., 2020). In the context of STEM education, bio- 
technology offers a unique approach to elucidating complex biological pro- 
cesses and fosters a holistic understanding of the natural world (Bahri et al., 
2014; de la Hoz et al., 2022). The significance of biotechnology in developing 
a society with a high level of scientific literacy has become essential due to its 
nature as a socio-scientific issue (Carver et al., 2017; Kidman, 2009; Dawson & 
Venville, 2009). Biotechnological advancements are revolutionizing various 
fields and posing challenges to conventional frameworks, exemplified by 
the use of genetically modified organisms (GMOs) to enhance agricultural 
productivity and the development of personalized medicine customized 
to individual genetic characteristics (Prokop et al., 2007; Sorgo & Ambrozic¢- 
Dolinsek, 2010; Sorgo et al., 2011). Therefore, it is imperative to have a well- 
informed society to effectively address the ethical, social, and environmental 
consequences associated with these advancements (Bybee, 2009; Hafte & 
Jemal, 2023). For example, in agriculture, biotechnology interventions have 
paved the way for genetically engineered crops that are resistant to pests, 
tolerate harsh environmental conditions, and have improved nutritional 
profiles (Malik & Chaudhary, 2019). Biotechnology has also revolutionized 
diagnostics, treatment methods, drug development, and an era of personal- 
ized medicine tailored to an individual's genetic makeup (Ahmed et al., 2013; 
Dwivedi et al., 2017). By immersing students in the study of biotechnology, 
the understanding of the transformative potential of science fosters an ap- 
preciation for the role of STEM subjects in solving real-world problems (Bahri 
et al., 2014; Movahedzadeh et al., 2012; Widiarti et al., 2022). 
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Biotechnology is central to advancing scientific literacy for several reasons and contributes to a comprehen- 
sive understanding of science and its applications in various fields (Casanoves et al., 2015; de la Hoz et al., 2022). 
Biotechnological innovations are ubiquitous in everyday lives, from genetically modified organisms in agriculture 
to personalized medicine in healthcare. Understanding these applications allows individuals to connect theoreti- 
cal knowledge with practical, everyday scenarios, developing a deeper understanding of the relevance of science. 
Researchers have implied that scientific literacy goes beyond knowledge of facts to understanding the ethical 
dimensions associated with scientific advances (Bybee et al., 2009; Prokop et al., 2007; Zoller et al., 2012). As biotech- 
nological innovations shape the future, a scientifically literate society is better positioned to meet the challenges 
and opportunities that arise (de la Hoz et al., 2022). Because biotechnology is at the forefront of addressing global 
issues such as food security, health disparities, and environmental sustainability, scientifically literate people can 
better contribute meaningfully to discussions and decisions impacting these critical areas. Biotechnology bridges 
the gap between theoretical knowledge and practical applications, promotes interdisciplinary thinking, develops 
critical thinking skills, addresses ethical considerations, prepares individuals for the future, cultivates curiosity, fosters 
global citizenship, creates employment opportunities, and demystifies the world of science for learners from diverse 
backgrounds. Determining individuals’ knowledge and attitudes toward biotechnology is, therefore, essential to 
having a scientifically literate society capable of understanding and contributing to the complex scientific landscape 
of the 21st century. 

In exploring students’ knowledge and attitudes toward biotechnology, it is essential to recognize that fostering 
scientific literacy is about imparting information and instilling attitudes and affective factors. Identifying, analyzing, 
and applying scientific knowledge is increasingly essential in an era characterized by abundant information. Bio- 
technology provides an ideal platform to cultivate these skills by linking concepts from the classroom to real-world 
scenarios. However, the effectiveness of education in preparing students for the biotechnology landscape depends 
on more than just teaching theoretical concepts. Exploring students’ attitudes toward biotechnology and assess- 
ing their receptivity to ethical dilemmas, social implications, and potential risks associated with biotechnological 
advances is necessary. Integrating biotechnology into STEM education should allow students enable them to think 
critically about the social implications of scientific advances. 

Biotechnology is a promising component of STEM education, but the success of its integration depends heavily 
on students’ perceptions and attitudes toward this field. Exploring these perceptions is critical for educators and 
policymakers alike, as it enables the identification of potential barriers to effective learning and provides informa- 
tion for strategies to increase engagement. In seeking this understanding, it is essential to consider the diversity of 
student populations and varying levels of engagement with biotechnology. Students from different demographic 
backgrounds, educational levels, and geographic locations may bring unique perspectives and experiences (de la 
Hoz et al., 2022; Erdogan et al., 2012; Sorgo et al., 2011). Consequently, a nuanced examination of student attitudes 
toward biotechnology should consider these contextual factors to provide a holistic view of the educational land- 
scape. Biotechnology, with its complicated mix of biological principles and technological applications, requires 
students to understand and apply basic concepts in new and evolving contexts. Therefore, examining their theo- 
retical understanding is necessary to assess students’ knowledge of biotechnology. In addition, the ethical aspects 
of biotechnology contribute to students’ even more complex understanding. Understanding the societal implica- 
tions, ethical considerations, and potential risks associated with biotechnology innovations is essential to educating 
students to be responsible and informed citizens. Hence, students’ knowledge and attitudes toward biotechnology 
are critical to raising a generation familiar with the complexities of modern science. 

From this perspective, scholars have conducted some studies to examine college students’ knowledge 
and attitudes toward biotechnology and its applications. For example, Bahri et al. (2014) determined biotechnol- 
ogy literacy based on the factors influencing students’ attitudes toward biotechnology. Their results showed that 
overall, students have an intermediate knowledge, perception, and attitude toward biotechnology. The study of 
Ozturk-Akar (2016) examined the knowledge and attitudes of science and non-science students toward biotech- 
nology applications. The results indicated that science undergraduates had higher knowledge and attitude scores 
than non-science students. However, both groups had sufficient knowledge of biotechnologies and no significant 
correlation between students’ attitudes and their knowledge of biotechnology. De la Hoz et al. (2022) studied the 
knowledge and attitudes of Swedish about biotechnology. Their results showed that pre-service elementary teachers 
had knowledge gaps regarding basic genetic concepts, and their attitudes toward biotechnological applications 
in health were positive but less toward purchasing and using genetically modified products. They also found a cor- 
relation between a higher level of knowledge and a more positive attitude. 

Among previous studies on students’ knowledge and attitudes toward biotechnology at the undergraduate 
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level, Prokop et al. (2007) found that Slovak college students had poor knowledge of biotechnological processes and 
had less favorable attitudes toward biotechnology, regardless of their understanding of genetic engineering. There 
was a reluctance to purchase genetically modified products due to negative attitudes towards the control of genetic 
engineering. Usak et al. (2009) found a significant correlation between the level of knowledge and attitudes towards 
biotechnology. Their study revealed a positive attitude towards agricultural biotechnology, although they had a 
limited understanding of biotechnology processes and negative attitudes toward genetically modified products. 

Sorgo and Ambrozié-Dolingek (2010) explored attitudes toward genetically modified organisms (GMOs) among 
prospective Slovenian teachers. The study uncovered that genetically modified plants and microorganisms, which 
were perceived advantageous, are generally embraced. However, there was uncertainty regarding organisms em- 
ployed for research or medical purposes and resistance to organisms used for food and enjoyment. Moreover, they 
detected a weak relationship between knowledge and attitude and knowledge and acceptance of GMOs. However, 
a strong relationship was noticed between attitude and acceptance. Erdogan et al. (2012) examined prospective 
teachers’ knowledge and attitudes toward biotechnology in four countries. Their results showed that prospective 
teachers’ knowledge about biotechnology was below average in all four samples. The results also showed significant 
differences in attitudes toward biotechnology among prospective teachers in the four countries. A study conducted 
by Prokop et al. (2013) aimed to explore the relationship between perceptions of modern risk technologies, risky 
behaviors, and neophobia towards genetically modified (GM) products. The study found that people who believed 
they were more vulnerable to infectious diseases had a more negative view of genetically modified (GM) products. 
Additionally, women were less likely to have a favorable attitude toward GM products. However, there was no con- 
nection between engaging in risky behaviors, women’s reproductive goals, and fear of new foods with attitudes 
toward GM products. 

Casanoves et al. (2015) conducted a study on the attitudes and knowledge of Spanish pre-service teachers 
toward biotechnology. The findings revealed that the teachers had a good understanding of the applications of 
biotechnology. Furthermore, the pre-service teachers opposed the acquisition of genetically modified products, 
supported the use of biotechnology for medical purposes, and expressed a desire to increase their knowledge of 
biotechnology and other scientific advancements. The study also found a positive correlation between a better un- 
derstanding of biotechnology and a positive attitude toward it. AbuQamar et al. (2015) researched the educational 
awareness, perceptions, and attitudes of university students in the United Arab Emirates toward biotechnology. 
The results showed that educational awareness of biotechnology knowledge and the environment was significantly 
related to the enrolled college and students’academic achievement. They also found a poor performance in students’ 
understanding. Research by Alanazi (2021) determined students’ and science teachers’ knowledge and attitudes 
toward biotechnology in Saudia Arabia. The results revealed limited awareness of biotechnology. 

China is among the countries that seek to leverage biotechnology to attain a high-income economy. However, 
the countries need to consider social acceptance and opinions regarding the issues emerging in this field. This is 
especially important as China has established itself as one of the primary national economic drivers in biotechnol- 
ogy. There is currently no available research on Chinese students’ attitudes and understanding towards biotech- 
nology. However, it is essential to note the importance of this research since it provides insight into the attitudes 
of students in STEM fields who will eventually become policymakers, leaders, and consumers and represent the 
future society of China. Previous research conducted with STEM students in other countries has revealed that they 
lack a comprehensive understanding of biotechnology, genetics, and genetically modified (GM) products. Studies. 
Additionally, these studies have found a weak but significant relationship between their content knowledge and 
attitudes related to the identified problems. However, no educational research has examined Chinese students’ at- 
titudes and understanding toward biotechnology. The existing research in the literature on attitudes towards and 
understanding biotechnology is mostly outdated, with data lacking from specific contexts such as Chinese STEM 
students. A notable gap exists in studies concerning the understanding and attitudes of STEM students towards 
biotechnology. This topic is of major significance as these students have a pivotal role in influencing the understand- 
ing and attitudes of future generations about biotechnology. This study examines STEM students’ knowledge and 
attitudes towards biotechnology to address this gap. Accordingly, the research question that guides this research 
is to examine the knowledge and attitudes of STEM students toward biotechnology. 

The results of this research offer insights into the effectiveness of current practices in STEM education in prepar- 
ing the next generation of scientifically literate individuals. The findings inform educators and policymakers and guide 
the development of curricula, pedagogical approaches, and outreach initiatives to cultivate a scientifically literate 
citizenry. By understanding the factors that shape students’ perceptions and knowledge gaps about biotechnology, 
educators can adapt their teaching strategies to improve engagement and understanding. 
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Research Methodology 
Background 


The authors used a survey method to answer the research question in this study. The quantitative research 
method uses a questionnaire as an instrument (Creswell & Creswell, 2017). For this aim, questionnaires were used 
to collect the data. The questionnaires included statements regarding knowledge and attitudes toward biotechnol- 
ogy. The questionnaires were administered to the participants in May 2023. According to the research question, 
the dependent variable was attitudes and knowledge about biotechnology applications. 


Participants 


The study involved a group of students selected from a single university in the Republic of China. A total of 
236 students (144 female, 61% and 92 male, 39%) were in grade 1 (106, 44.9%), grade 2 (41, 17.4%), grade 3 (31, 
13.1%), grade 4 (34, 14.4%), and master (24, 10.2%). These students were selected from different parts of the country 
based on their performance on university entrance exams. While the universities in this study may not be nationally 
representative, it is worth noting that the students came from diverse social and cultural backgrounds, making 
them typical of public universities. It is important to indicate that the purpose of this study is not to generalize 
the findings to all students in the country but rather to serve as a starting point for future nationwide research. All 
participants had taken at least one course in STEM subjects at different levels. The participants were involved in 
the research voluntarily. In addition, the authors considered the ethical issues with the participants’ participation 
in the research in data collection and analysis. 


Data Collection Instruments 


The participants’ knowledge of biotechnology was assessed using the Biotech XXI questionnaire developed 
by Casanoves et al. (2015). The Biotech XXI questionnaire is based on a framework that covers various subjects 
regarding genetically modified organisms and biotechnology. It consists of 21 questions that test the participants’ 
knowledge, with response options of “true”, “false,"and “do not know.’ The primary objective of the questionnaire is 
to evaluate the participant's level of understanding of biotechnology, particularly in terms of genetic knowledge 
and the practical applications of biotechnology. Many researchers have used it to assess individuals’ knowledge 
of biotechnology in previous studies (Casanoves et al., 2015; de la Hoz et al., 2022). 

To administer the questionnaire to the participants, a standardized English version of the original question- 
naire was used to produce a Chinese translation. A second independent back-translation was performed in English 
to confirm the accuracy of the translation. The final questions were revised and then evaluated with a sample of 
20 participants to determine the readability and comprehensibility of the questionnaire. The biotechnology at- 
titude questionnaire is a 28-item survey developed by Erdogan et al. (2009). It comprises seven dimensions that 
measure attitudes towards biotechnology and its applications. The questionnaire consists of seven key factors that 
assess individuals’ attitudes toward biotechnology and genetically modified organisms. The factors are related to 
the consumption of GM products, the use of GM in the agro-industry, public awareness of GMOs, shopping of GM 
products, ethics of genetic modifications, the ecological impact of genetic engineering, and the use of genetic 
engineering in human medicine. This reliable and valid research tool makes it useful for examining participants’ 
attitudes toward biotechnology. Numerous researchers (e.g., Alanazi, 2021; Erdogan et al., 2012; Ozturk-Akar, 2016) 
have employed it to evaluate participants’ attitudes toward biotechnology and genetically modified organisms. 


Data Collection 


The researchers distributed a paper questionnaire to students in their class. This created a controlled environ- 
ment where all participants received the same instructions. The data was gathered from a group of individuals 
enrolled in a publicly-funded research institution in the Republic of China. All the participants in the study had 
completed STEM coursework. The researchers made sure that the questionnaire was administered consistently. 
Moreover, they ensured that the ethical standards of each respective country were followed. 

When conducting research studies, it is crucial to consider the reliability of the questionnaire used. In this 
study, the Biotech XXI and BAQ instruments showed satisfactory results regarding their first internal consistency, 
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with Cronbach's alpha scores of 0.87 and 0.71, respectively. Data collected during the study was used to determine 
the subdimensions reliability scores, their corresponding abbreviations, and the number of items in each subscale 
to evaluate their reliability. The subscale titles are abbreviated and referenced throughout the text for clarity. Table 
1 displays subdimensions’ mean and standard deviation and reliability scores, their corresponding abbreviations, 
and the number of items inside each subscale. 

The study found that the Biotech XXI and BAQ instruments used to measure participants’ scores demonstrated 
satisfactory results, with Cronbach's alpha of .87 and .71, respectively, indicating good internal consistency. The in- 
struments reliability was further analyzed using participant data, revealing a Cronbach's alpha of .87 for the Biotech 
XXI and .71 for the BAQ. The reliability analysis was conducted for 7 subdimensions, with the Cronbach's alpha for 
the BAQ subscales ranging from .70 to .80. Table 1 below presents the reliability values, abbreviations, and number 
of items per subdimensions. The abbreviations for each subdimension are used throughout the rest of the text. 


Table 1 
The Results Regarding Each Factor 


Number of the Reliability 


Factor names M SD tainé (a) 
Factor 1. Consumption of GM products (CGMP) 3.03 0.94 4 80 
Factor 2. GM in Agro-Industry (GMAI) 3.49 0.66 5 1 
Factor 3. Public awareness of GMO (PAGMO) 3.03 0.60 3 12 
Factor 4. Shopping of GM products (SGMP) 2.74 0.70 6 19 
Factor 5. Ethics of genetic modifications (EGM) 2.71 0.84 3 70 
Factor 6. Ecological impact of genetic engineering (EIGE) 3.10 0.66 4 10 
Factor 7. Use of genetic engineering in human medicine (UGEHM) 3.55 0.91 3 77 
Whole instrument 28 afd 


Research Results 
Knowledge of Biotechnology 


The questionnaire included 21 questions testing biotechnology knowledge. The obtained Cronbach's alpha 
coefficient was .87, indicating that the research had acceptable reliability. Table 2 shows results according to the 
percentage of respondents who responded correctly and incorrectly and did not know the responses. Most partici- 
pants correctly answered the first 12 questions and question 14 with more than 50% accuracy. On the other hand, 
the remaining 8 questions were mostly answered incorrectly or marked as“do not know” by over 50% of participants. 

Based on the findings in Table 2, it can be inferred that the participants lacked sufficient knowledge about 
various fundamental concepts related to genetics and biotechnology, especially those related to GMOs. For ex- 
ample, most students (86%) gave incorrect answers or were unaware that DNA is chemically identical in all living 
organisms. In addition, a mere 31% of the students recognized that GMOs are not inherently larger than ordinary 
organisms, and many students expressed uncertainty over the presence of toxic substances in GMOs. However, 
the respondents demonstrated a good understanding of certain biotechnology applications, such as the use of 
bacteria in the production of cheese and vinegar (80% of students knew this) and the modification of genetic 
characteristics against disease, nutritional value, or productivity (64% of students were aware of this). 

To determine the difference between genders in terms of knowledge about biotechnology, a t-test for inde- 
pendent samples was performed. The results showed no significant differences [t ,,, = .204, p > .05] between the 
average mean score of females (M = 2.02, SD = .35) and males’ mean score (M = 2, SD = 0.44). Based on this result, 
no significant differences existed between genders. In addition, one-way ANCOVA was performed to examine 
the effect of grade level on students’ knowledge of biotechnology. The main effect of grade level [F |, ,,,. = 5.81, 
p < .05]. Significant differences were found between the scores from grades one-third, two-third, third-four, and 
four and master students. 
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Attitudes toward Consumption of GM Products 


Participants showed neutral attitudes toward the consumption of GM products. A significant difference 
was found between female students’ mean scores (M = 3.22) and males’ mean scores (M = 2.72) [t 534 = 4-129, 
p < .05]. Based on this result, it can be concluded that gender has a significant effect on the consumption of 
GM products. The results show that female students had higher mean scores than male students. The effect 
size (n2 = .91) calculated as a result of the test showed that this difference was moderate. In particular, females 
and males indicated positive attitudes on item 10 about not giving GM food to children. This item's mean score 
was higher than the other three items in this subdimension. In addition, females had higher mean scores about 
altering the genes in fruits to improve their taste and against altering the genes of fruits and vegetables and the 
risks of using genetically modified food. In addition, one-way ANCOVA was performed to examine the effect of 
grade level on students’ attitudes in this dimension. The main effect of grade level [F (4,231) = 2:66, p < 0.05] was 
found significant. These significant differences were found between the scores from third to fourth grades. The 
mean scores of third grades were higher than those of fourth grades. 


Attitudes toward GM in the Agro-Industry 


All items’ scores for females and males exceeded 3, indicating that students in the Agro-industry hold 
neutral attitudes about GM. Participants showed higher attitudes than neutral regarding the use of GM in the 
Agro-industry. A significant difference was not found between female students’ mean scores (M = 3.49) and 
males’ mean scores (M = 3.5) [t 534 = 7-06, p > .05]. Based on this result, it can be concluded that gender has no 
significant effect on the use of GM in the Agro-industry. The results show that female and male students had 
similar scores regarding the items in this subdimension. In particular, both females and males indicated positive 
attitudes about the items related to agreeing with the use of genetic engineering in the therapy of genetically 
determined diseases and wanting to know more about genetically engineered foods. These items’ mean scores 
were higher than the other three items in this subdimension. The results indicate that male and female par- 
ticipants held positive attitudes about using genetic engineering to treat genetically determined diseases and 
wanted to learn about genetically modified foods. Furthermore, a one-way ANCOVA was conducted to explore 
the impact of grade level on students’ perceptions in this particular aspect. The main effect of grade level [F é 
331) = 1-17, p >.05] revealed no significance among grade levels. 


Attitudes toward Public Awareness of GUO 


Participants showed neutral attitudes toward public awareness of GMOs. The average mean score for the 
three items in this dimension was 3.03. No significant difference was found between female students’ mean scores 
(M = 3.05) and males’ mean scores (M = 3.00) [t 234 =.693, p > .05]. Based on this result, it can be concluded that 
gender has no significant effect on the attitudes toward public awareness of GMOs. The results show that female 
and male students had similar mean scores. In particular, females (M = 3.67) and males (M = 3.65) indicated posi- 
tive attitudes on item 25 about trusting the food industry to take necessary actions to provide safe genetically 
engineered foods. This item’s mean score was higher than the other two items in this subdimension. For the 
other two items, females and males had similar mean scores. Specifically, the average scores for both females 
and males were below the mean scores by approximately 3 points. This suggests that the existing governmen- 
tal regulations effectively protect the public from the risks linked to genetically engineered foods and that the 
public is adequately informed about these risks. In addition, the effect of grade level on students’ attitudes 
toward public awareness of GMOs was examined through one-way ANCOVA analysis. Accordingly, the main 
effect of grade level [F 4,231) = 1-97, p >.05] was found not significant. This result shows there are no statistical 
differences among grade levels. 


Attitudes toward Shopping for GM Products 


The average mean score for the three items in this dimension was 2.74. This mean score was less than 3, 
referring to students’ neutral attitudes toward shopping for GM products. A significant difference was found 
between female students’ mean scores (M = 2.64) and males’ mean scores (M = 2.9) [t 934 = 7-284, p < .05]. Based 
on this result, it can be concluded that gender significantly affects the shopping of GM products. The results 
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show that male students had higher mean scores than female students when shopping for GM products. The 
effect size (n2 = .69) calculated as a result of the test showed that this difference was moderate. Participants 
showed only neutral attitudes toward an item related to eating genetically modified tomatoes. The lowest mean 
scores were found for the item related to inserting genes from human cells into the fertilized eggs of sheep. The 
other four items were found to have the lowest mean scores (less than 3 mean scores) related to the effects of 
GM food on human health, thinking about GM products having a better taste, buying the GM product, and sup- 
porting changing the genes in cattle to make their meat more nutritious. In particular, males had more positive 
attitudes than females for all six iterms in the subdimension. In addition, one-way ANCOVA was performed to 
examine the effect of grade level on students’ attitudes in this dimension. The main effect of grade level [F eax 
= 4,42, p < .05] was found significant. The effect size (n2 = .07) calculated as a result of the test showed that this 
difference was moderate. These significant differences were found for grades 1-4, 2-3, and 3-4. The mean scores 
of the fourth grade were higher than the other grades. 


Attitudes Toward Ethics of Genetic Modifications 


The mean score for the three items in this dimension was 2.71, on average. The mean score was below 3, 
indicating that students held neutral attitudes about the ethics of modifications to genes. A significant difference 
was found between female students’ mean scores (M = 2.81) and males’ mean scores (M = 2.55) [t 934 = 2:6, P < 
.05]. Based on this result, it can be concluded that gender significantly affects the ethics of genetic modifica- 
tions. The results show that female students had higher mean scores than male students. The effect size (n2 = 
.84) calculated as a result of the test showed that this difference was moderate. In particular, females had neutral 
attitudes on item 1 about transferring genetic material between plants and animals. This item’s mean score was 
higher than the other two items in this subdimension. In addition, females had higher mean scores than males. 
Based on the results for the other two items in this subdimension, the participants indicated that DNA manipula- 
tion is unethical and human beings do not have the right to intervene in DNA; it is against nature. In addition, 
one-way ANCOVA was performed to examine the effect of grade level on students’ attitudes in this dimension. 
No significant differences regarding the effect of grade level [F , = 1.41, p > .05] were found. Thus, there are 
no significant differences in grade level. 


4,231) 


Attitudes toward Ecological Impact of Genetic Engineering 


Participants showed neutral attitudes toward the ecological impact of genetic engineering. The average 
mean score for the three items in this dimension was 3.10. A significant difference was found between female 
students’ mean scores (M = 3.15) and males’ mean scores (M = 3.01) [t 934 = 1-585, p > .05]. Based on this result, 
it can be concluded that gender has no significant effect on the ecological impact of genetic engineering. The 
results show that female students had higher mean scores than male students, but this difference is insignificant. 
Both females and males expressed favorable attitudes towards item 23, which pertains to the potential hazard 
of hybridization between genetically modified and regular plants, posing a risk to the original genetic resources 
of wild plants. This item’s mean score was higher than the other three items in this subdimension. The lowest 
mean score in this subdimension was found to ban the production and purchase of genetically engineered 
products. Regarding the remaining two aspects of this dimension, the participants were neutral about refraining 
from modifying plant genes to enhance oil production for industrial purposes and the potential disruption of 
ecological linkages caused by genetic manipulations. In addition, one-way ANCOVA was performed to examine 
the effect of grade level on students’ attitudes in this dimension. The main effect of grade level [F (4,231) = 2-67, P 
< 0.05] on the attitudes toward the ecological impact of genetic engineering was found to be significant. These 
significant differences were found only between the scores from grades 1 and 4. The average scores for fourth- 
grade students were higher than those for first-grade students. 


Attitudes Toward Use of Genetic Engineering in Human Medicine 


Participants showed more positive attitudes toward using genetic engineering in human medicine. The 
average mean score for the three items in this dimension was 3.55. This mean score had the highest score in 
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all subdimensions. No significant difference was found between female students’ mean scores (M = 3.53) and 
males’ mean scores (M = 3.57) [t ,,, = -.29, p > .05]. Based on this result, it can be concluded that gender has no 
significant effect on using genetic engineering in human medicine. In particular, females and males indicated 
positive attitudes on item 8 about using genetic engineering for nonfood purposes, such as the production of 
human medicines. This item’s mean score was higher than the other two items in this subdimension. Partici- 
pants had favorable attitudes about using GM microbes to decompose human sewage and insulin production 
using genetically modified microbes for the other two items. In addition, one-way ANCOVA was performed to 
examine the effect of grade level on students’ attitudes in this dimension. The main effect of grade level [F aaa 
= .44, p > 0.05] on the use of genetic engineering in human medicine was not significant. Thus, no significant 


differences were found in grade level. 


Table 2 
Results of Knowledge of Biotechnology 


Correct Responded Responded penot 
nme answer earrecly incorrectly (%) mee 
(*) ye (%) 
1 AIDS is a genetic disease False 83 6 11 
2 Bacteria are used in the elaboration of daily products (cheese, vinegar, True 80 6 14 
vitamin C) 
3 Only when we eat GM food, we eat genes False 64 4 32 
4 — Inour body, there are more bacteria than people in the world True 70 t 23 
5 Agood hygiene helps to prevent genetic diseases False 69 13 18 
6 Children resemble their parents because they share the red blood cells False 63 6 31 
7 _ Itis possible to change the genetic characteristics of a plant to make it True 64 8 28 
more resistant to a given plague 
8 Mutations are only possible by genetic manipulation in the laboratory False 76 9 15 
9 Ahigh production of vitamins by a fruit is only possible by genetic False 56 15 29 
manipulation of that fruit 
10 Through genetic modification, foods with higher nutritional values can True 62 9 29 
be achieved 
11. Microorganisms are used to purify sewage True 64 6 30 
12 Ayoghurt is a biotechnological product True 59 13 28 
13 Insulin is obtained by the use of genetically modified (GM) bacteria True 25 16 59 
14 Genetic material exchange between different species is only possible by False 52 14 34 
manipulation in the laboratory 
15 The most powerful toxic substances are naturally occurring True 46 23 31 
16 GMOs have a high number of toxic substances False 37 29 34 
17 Chemically, the genetic material (DNA) is identical in all the organisms True 14 55 31 
18 In the kidney cells genome, you can also find information about the True 34 13 ie) 
color of your hair 
19 Genetically modified organisms (GMOs) are larger than normal False 31 24 45 
20 Crocodiles have the same genetic material as ostriches True 15 25 60 
21 AGMO is always a transgenic False 20 15 65 
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Correlations between Knowledge and Attitudes 


A multivariate partial correlation was performed while holding the grade level variable constant to examine 
the correlation between knowledge of biotechnology and subdimensions of attitude toward biotechnology. The 
findings presented in Table 3 suggest that out of 28 correlations, 16 are statistically significant and exceed 0.15. 
Among these significant correlations, six were observed to be negative. Four correlations linked to knowledge 
were not found to be significant. Notably, significant correlations were identified between knowledge and Ethics 
of genetic modifications (EGM) [r (233) = .243, p < .01], Use of genetic engineering in human medicine (UGEHM) 
[r (233) =-.229, p < .01], GM in Agro-Industry (GMAI) [r (233) = -.330, p < .01]. The association between knowledge 
and GM in Agro-Industry exhibited a moderate connection, while the remaining correlations were very modest. 
In summary, these findings indicate that possessing knowledge of biotechnology was partially associated with a 
more favorable attitude toward biotechnology. 


Table 3 
Results of Correlation Analysis between Knowledge and Attitudes 


1 2 3 4 5 6 7 8 
Consumption of GM products (CGMP) - 
Shopping of GM products (SGMP) -.572"* 
Ethics of genetic modifications (EGM) .108** -.350** 
ee impact of genetic engineer- 604" 434" Ago" . 
Use of genetic engineering in human 113 461* 083 451" 
medicine (UGEHM) 
GM in Agro Industry (GMAI) -.107 .298** -.238** 015 .638** 
Public awareness of GMO (PAGMO) -.027 348** .065 -.054 .084 .153* - 
Knowledge .088 .086 .243** -.089 -.229** -.330"* 074 


** Correlation is significant at the .01 level (2-tailed). 
*_ Correlation is significant at the .05 level (2-tailed). 


Discussion 


The study found that many participants lacked an understanding of basic concepts in genetics and biotech- 
nology, with a majority providing incorrect answers or being unaware of the universality of DNA among living 
organisms. Additionally, merely 31% of the students knew that GMOs do not inherently possess a bigger size than 
ordinary organisms, and a significant number of them were uncertain about the presence of toxic substances in 
GMOs. The participants exhibited a commendable comprehension of specific biotech practices, such as leveraging 
bacteria in the production of vinegar and cheese and manipulating genetic traits to bolster an organism's resistance 
to illnesses, nutrient content, or efficiency. (64% of students were aware of this). In addition, the results showed no 
significant differences between female and male participants and some significant differences among grade levels. 

These findings are consistent with those of previous research (AbuQamar et al., 2015; Erdogan et al., 2012; 
Lamanauskas & Makarskaite-Petkeviciene, 2008; Ozturk-Akar, 2016; Turkmen & Darcin, 2007; Usak et al., 2009), 
which found that participants at the college level did not have satisfactory levels and reported a weak knowledge 
of biotechnology. According to a recent study by De la Hoz et al. (2022), Swedish pre-service teachers demonstrated 
agood understanding of comprehension of genetic disease and biotechnology applications. However, they scored 
lower on questions related to basic concepts of genetics and GMOs. These findings are consistent with those of 
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Casanoves et al. (2015). The results between gender and knowledge are not similar to those of (Erdogan et al., 
2012), who have significant differences in biotechnology knowledge regarding gender variables. 

The study suggests that participants have difficulty with complex biotechnologies due to a lack of genetic 
knowledge. The study's results indicate that the lack of knowledge found in participants still seems to exist among 
undergraduate participants compared to the results of previous studies conducted with undergraduate participants. 
This result suggests a lack of biotechnology literacy. The study found that the participants held a neutral attitude 
towards biotechnology, with mean scores ranging from 2.71 to 3.55. They were generally opposed to purchasing 
GM products but supported using biotechnology for medical purposes. Furthermore, they strongly desire to expand 
their knowledge about GM products. The participants did not have a negative opinion of biotechnology in general 
but were critical of its purpose and use. The discovery, as mentioned above, indicates the presence of an inherent 
utility-value framework that is exerting an impact on the alteration of perspectives about the implementation of 
genetically modified organisms (GMOs), contingent upon the intended purpose of this technology. Therefore, 
the use of GMOs should be acceptable if they can save human lives or prevent diseases, but not if they are used 
solely for economic advantage. These findings are consistent with those of previous studies (Erdogan et al., 2012; 
Ozturk-Akar, 2016; Ozel et al., 2009; Prokop et al., 2007; Prokop et al., 2013; Sorgo et al., 2011; Usak et al., 2009). 

Participants displayed a neutral attitude toward consuming genetically modified (GM) products. A significant 
difference was found in favor of female participants. This result is inconsistent with those of de la Hoz et al. (2022), 
who found that preservice teachers had opponent's attitudes toward buying GM products. Also, the results are not 
inconsistent with Erdogan et al. (2012) and Prokop et al. (2007), who found lower attitudes about consuming GM 
products. The findings regarding gender differences are not similar to those of Erdogan et al. (2012) and Prokop 
et al. (2007). They found that male participants’ attitude scores had higher attitudes than female undergraduates. 

The discrepancies observed between previous research and the present study regarding the consumption 
of GMPs may have arisen from varying attitudes prevalent in different countries. The educational approaches, 
backgrounds, and discourses in the countries’ media may have influenced the participants’ attitudes toward GMPs. 
Differences in participants’ perception regarding the level of risk associated with consuming genetically engineered 
products could be attributed to the different attitudes among participants toward the same dimensions and items 
(Erdogan et al., 2012; Usak et al., 2009). Moreover, the presence of biotechnology goods (Bailey & Lappé, 2002; 
Prokop et al., 2007; Ozel et al., 2009) may serve as an additional factor contributing to these variations. 

Students had neutral attitudes toward GM in the agro-industry. The scores were higher than 3 mean points. In 
particular, males and females had positive attitudes towards genetic engineering for treating genetically determined 
diseases and learning more about genetically engineered foods. In this study, the positions of the participants were 
found to be similar to those in previous studies conducted with preservice teachers, such as the studies conducted 
by Erdogan et al. (2012) and Massarani and Moreira (2005), and Ozturk-Akar (2016). The results of these studies in- 
dicate that a majority of participants were in favor of using genetic engineering for treating genetically determined 
diseases. For example, De la Hoz et al. (2022) conducted research showing that 71% of preservice teachers favored 
using GMOs for medical purposes and disease studies. Additionally, three out of four respondents approved of 
using genetic studies to develop medicine. A significant difference was not found between genders. In addition, 
grade level had no significance among grade levels. 

Participants showed neutral attitudes toward public awareness of GMOs. No significant difference was found 
between both genders. It was found that the participants had lower attitudes towards three specific items, which 
suggests that they believe the government regulations are adequate for protecting the public from the risks associ- 
ated with genetically engineered foods. Additionally, the participants believe the public is already well-informed 
about these risks. These findings differ from those of Erdogan et al. (2012) and Usak et al. (2009), who came to 
different conclusions about the effectiveness of governmental regulations. The lower attitudes are similar to those 
of Ozturk-Akar (2016), who found lower attitudes about public awareness regarding GMOs. They found positive 
results about governmental regulations. On the other hand, the findings about sufficiently informing the public 
about the risks associated with genetically engineered foods are similar to those of Erdogan et al. (2012). A lack of 
knowledge regarding biotech procedures and a distrust of governmental regulations and oversight of GEFs may 
have caused this. In addition, the grade level did not impact the attitudes in this dimension. 

The attitudes toward shopping for GM products had lower than three mean scores. Male students had statisti- 
cally higher mean scores than female students when shopping for GM products. Participants showed only neutral 
attitudes toward an item related to eating genetically modified tomatoes. The lowest mean scores were found for 
the item related to inserting genes from human cells into the fertilized eggs of sheep. Grade levels had a significant 
impact on the attitudes in this dimension. This result is similar to the findings of Erdogan et al. (2012), Ozturk-Akar 
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(2016), and Usak et al. (2009). In previous studies, participants hesitated to consume genetically modified products 
and were unwilling to purchase them (Erdogan et al., 2012; Ozturk-Akar, 2016; Usak et al., 2009). The participants 
also had the lowest mean scores (less than 3 mean scores) related to the effects of GM food on human health, 
thinking about GM products having a better taste, buying the GM product, and supporting changing the genes 
in cattle to make their meat more nutritious. These results confirm the results of Erdogan et al. (2012), Ozturk-Akar 
(2016), and Usak et al. (2009). The participants’ fear and distrust of GM products may stem from potential dangers, 
such as an assumed link between GMOs and cancer [32]. The fourth grade was higher than the other grades. 

Students’ attitudes toward the ethics of genetic modifications had lower mean scores. Female students had 
significantly higher mean scores than male students. The participants indicated that DNA manipulation is unethical, 
and humans do not have the right to intervene in DNA. This result is partly similar to Erdogan et al. (2012) and Usak 
et al. (2009). No significant differences regarding the effect of grade level. Participants showed neutral attitudes 
toward the ecological impact of genetic engineering. The female students had significantly higher mean scores than 
male students. This result is different from the study of Erdogan et al. (2012) and Usak et al. (2009). All participants 
displayed favorable attitudes regarding the risk of hybridization between genetically modified and non-modified 
plants, potentially harming the original genetic resources of wild plants. The mean scores were lowest regarding 
prohibiting genetically modified product manufacturing and purchase. In addition, the participants expressed a 
lack of bias towards modifying the genes in plants to enhance their oil production for industrial purposes, as well as 
concerns about the potential disruption of ecological interactions resulting from genetic modifications. Moreover, 
their grade level highly influenced the participants’ perceptions of the ecological impact of genetic engineering. 
The average scores of grade 4 participants outperformed those of grade 1 participants. The participants in the study 
expressed positive attitudes towards the use of genetic engineering in human medicine. The mean score for this 
subdimension was the highest among all the subdimensions, although no significant difference was found. The 
participants also showed favorable attitudes towards using genetic engineering for non-food purposes, such as 
producing human medicines. They also expressed positive attitudes towards using genetically modified microbes 
to decompose human sewage and produce insulin. The study found that the participants’ grade level did not 
significantly affect their attitudes toward genetic engineering. 

The correlation study yielded both statistically significant and insignificant findings concerning knowledge. 
Four associations related to knowledge did not show statistical significance. Strong relationships were observed 
between knowledge and ethics, human medicine, and agro-industry. The association between knowledge and 
agro-industry exhibited a moderate level, whereas the remaining relationships showed relatively low levels. Col- 
lectively, these findings indicate that possessing expertise in the field of biotechnology is associated, to some 
extent, with having more favorable attitudes toward biotechnology. Previous studies have shown that having a 
better knowledge of biotechnology leads to a more positive attitude (e.g., de la Hoz et al., 2022; Ozel et al., 2009; 
Prokop et al., 2007; Usak et al., 2009). However, our study yielded different results. Despite having little knowledge 
of biotechnology, the participants had a positive attitude toward the biotechnological applications mentioned 
in the study, such as human medicine. This is in agreement with the conclusions drawn by de la Hoz et al. (2022), 
Prokop et al. (2007) and Usak et al. (2009). Furthermore, the correlation analysis revealed a weak association between 
participants’ knowledge about biotechnology and their attitude towards biotechnology applications, contradict- 
ing prior study findings. Hence, discerning the lack of a meaningful correlation between participants’ knowledge 
of biotechnology and their attitudes toward biotechnology applications is challenging, similar to the findings of 
Ozturk-Akar (2016) and Sorgo et al. (2011). 


Conclusions and Implications 


STEM students need to have adequate knowledge and positive attitudes about biotechnology as they play a 
crucial role in shaping the understanding of future generations regarding the impact of biotechnology on society. 
Personal, social, and cultural background factors may likely influence participants’ attitudes toward biotechnology, 
independent of their level of education. However, education continues to be one of the most important determi- 
nants. Studies indicate a difference in attitudes towards biotechnology between younger students (12-year-olds) 
and older students (17-year-olds), with the former exhibiting less favorable attitudes. This underscores the signifi- 
cance of incorporating biotechnology concepts into the curriculum during the early stages of school. To promote 
biotechnological literacy among the participants, it is crucial to possess basic knowledge about biotechnology and 
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have a positive attitude toward it. Although some participants may not be fond of science and technology and their 
practical applications, a positive attitude towards these subjects is essential for having a science literacy society. 
This study establishes an empirical basis for creating instructional methods focusing on attitudes and knowl- 
edge for students studying STEM in biotechnology. An effectively structured curriculum can establish a firm basis, 
but its subsequent enhancement and advancement ultimately depend on policy decisions and implementation 
strategies. Hence, raising participants’ knowledge and attitudes about biotechnology is crucial to elevate the 
education standard. This research suggests that formal education alone is insufficient for STEM students to acquire 
scientific literacy about biotechnologies. The scientific literacy of science students in biotechnologies cannot be 
confirmed either. Additionally, the study revealed no significant correlation between participants’ knowledge 
of biotechnology and their views toward biotechnology applications. The research was conducted on a group 
of individuals in a college setting. However, to achieve more extensive results within a nationwide context, ad- 
ditional research is required to assess the suitability of these findings. What is lacking in this current study is the 
crucial consideration of participants’ informal learning experiences and the importance of public presentation of 
biotechnology applications in China when concluding their knowledge and attitudes towards biotechnologies. 


Recommendations 


The future occurrence of the legalization of genetically modified products is anticipated. Hence, people and 
society must comprehensively understand this subject matter since research suggests increased knowledge cor- 
relates with more favorable perceptions of biotechnology (de la Hoz et al., 2022; Usak et al., 2009). The results of 
this study indicate that incorporating biotechnology messages within the STEM curriculum may not sufficiently 
equip students with the necessary knowledge and skills in the biotechnology field. The limited comprehension of 
biotechnology processes may stem from a superficial comprehension of biotechnology as presented in educational 
materials and media. A more comprehensive comprehension of biotechnology can be achieved by critically exam- 
ining the STEM curriculum and engaging in public discourse with scientists, potentially facilitated through media 
platforms such as television or publications. The study reveals that attitudes towards biotechnology are generally 
unfavorable when it comes to the acquisition of genetically modified products. This finding holds significant im- 
plications for future food policy in many nations. Providing additional sources of information about biotechnology 
can facilitate the acquisition of a more profound comprehension and heightened awareness of this field among 
students and the general public. Moreover, educators can employ this knowledge to enhance their instruction on 
genetic engineering. Simultaneously, it is imperative to acknowledge the significance of adequately equipping 
STEM educators to teach biotechnology effectively. However, further study and scholarly inquiry are warranted to 
deepen our understanding of this subject matter. 


Limitations 


The present study is subject to many limitations. First, the present study comprised a group of participants 
drawn from a suitable sample of individuals enrolled in a university setting. Further research should be undertaken 
to assess these findings’ applicability and gain broader outcomes within a national framework. Second, the study 
employed well-established instruments for data collection, and the findings gained in this study hold the potential 
to yield generalizable conclusions for the existing body of literature. Nevertheless, using nonrandomized sampling 
limits the research design and the comparatively limited sample size. It is impossible to assert the generalizability 
of the findings to all students in the studied country; nonetheless, the results offer some preliminary insights. 
The selection of students at the university where the present study was done is based on a national examination. 

Consequently, a standardized curriculum is implemented throughout all universities within the nation. The 
students in these programs exhibit comparable demographic traits, rendering the selected cohort a suitable rep- 
resentation. Furthermore, this study presents remarkable findings from many educational and cultural contexts. 
Hence, it would be stimulating for future inquiries to broaden the study's scope to assess the applicability of the 
results in diverse educational and cultural contexts. When evaluating participants’ understanding and attitudes 
regarding biotechnologies, it is crucial to consider the influence of their informal learning experiences and the 
importance of public demonstration of biotechnology applications. 
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Abstract. /n Taiwan, the education system 
places a strong emphasis on higher 
education. Students who perform well 

in the National High School Entrance 
Examination are given priority admission to 
public high schools, while those with lower 
scores tend to enroll in vocational schools. 
It’s worth noting that students with low 
scores in this examination often struggle 
with reading and natural literacy, typically 
scoring below level 2 in PISA assessments. 
This study examined the impact of 
implementing the mobile platform 

OKAR reading strategy on students’ 
comprehension of popular science reading 
and their logical thinking abilities. The 
research was conducted with two classes of 
first-year vocational school students. 

The findings indicate that when students 
used the OK4R mobile platform for 
popular science reading, it hada 

positive effect on their popular science 
reading comprehension and their logical 
thinking skills. Notably, female students 
outperformed their male counterparts. The 
study also revealed that several factors, 
including gender, basic abilities, and other 
variables, influenced the effectiveness of 
the OK4R mobile platform for vocational 
students. Those with lower science scores 
appeared to benefit the most from this 
approach. 
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Introduction 


The 2018 PISA student performance report from the OECD indicates 
that Taiwanese students made slight progress overall. However, there has 
been a significant increase in the proportion of students achieving high levels 
(level 5 and above), while the performance of low-achieving students (level 
2) has not seen significant improvement over the past decade. This lack of 
improvement for low-achieving students has been a persistent issue in Taiwan 
(Crato, 2021). The PISA report also highlights gender differences in academic 
performance. Girls tend to outperform boys in reading literacy, while boys 
generally do better than girls in understanding scientific passages. One pos- 
sible explanation is that boys may have a stronger background in scientific 
content, leading to higher overall science scores (Lee, 2015). Research on 
Polish vocational students suggests that their reading literacy is lower than 
that of high school students. After implementing teaching interventions, 
girls tend to demonstrate greater improvements in reading skills compared 
to boys, and their scientific skills also show more improvement than those 
of boys (Tereszewski & Walczynski, 2015). Conversely, studies by Sanchez 
and Wiley (2010) have shown that traditionally, boys have outperformed 
girls in visual-spatial and overall science learning, which can have an impact 
on career decisions. However, further research has shown that the use of 
illustrations and animation effects can effectively reduce these gender dif- 
ferences. Additionally, Ghazivakili et al. (2014) found that learning style, age, 
and gender are related to the learning environment. Girls tend to excel in 
inference and critical thinking skills, but there are no significant gender dif- 
ferences in evaluation, analysis, inductive reasoning, or deductive reasoning. 
Overall, the studies mentioned highlight the intricate interaction of various 
factors, including gender, teaching methods, and learning environments, in 
influencing students’ academic performance in reading and science. 
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Yore and Denning (1989) identified three key abilities in scientific reading: vocabulary ability, comprehension 
ability, and discussion ability. The importance of popular science courses and popular science reading is under- 
scored by Guthrie et al. (2000), who found that the effective use of reading strategies can enhance students’ intrinsic 
motivation for scientific reading. Several strategies and teaching methods have been found to improve students’ 
reading motivation, learning ability, and the effectiveness of comprehending scientific texts or popular science 
articles. These strategies include the application of reading comprehension strategies, the use of multimedia ele- 
ments like animations and images, questioning strategies, and interactive teaching methods (Baird-Thompson, 
2023; Barak et al., 2011; Dalacosta et al., 2009; Mason et al., 2013). Ozuru et al. (2009) noted a positive correlation 
between integrated comprehension ability and students’ prior knowledge, indicating that higher reading ability 
benefits from reading highly cohesive texts. Akin et al. (2015) went further to find that reading scientific texts not 
only enhances academic performance but also significantly improves critical thinking and critical reading skills. In 
practical teaching, educators must pay close attention to students’ reading processes, especially in the context of 
popular science. Using reading strategy tools to facilitate interaction with the text and construction of knowledge 
can be a valuable approach to effectively employ auxiliary tools and enhance the overall reading experience 
(Jamshidifarsani et al., 2019; Ponce et al., 2020; Wang et al., 2010). 

Logical thinking involves the process of using concepts, judgments, and reasoning to reflect on reality during 
the acquisition of knowledge. The ability of individuals to set independent goals or beliefs is considered a funda- 
mental aspect of the development of formal thinking. Structuring logical thinking abilities in childhood is crucial 
for future cognitive development (Handley et al., 2004). Courses and educational materials, particularly popular 
science books that align with the school curriculum or cater to the age and level of students, can gradually cultivate 
and establish students’ logical thinking. These resources play a vital role in enhancing mathematical logic skills and 
serve as valuable supplementary teaching materials alongside formal science education (Dewi & Hulyadi, 2015; 
Hodijah et al., 2018; Sadi & Cakiroglu, 2015). Research by Tobin and Capie (1981) employed the Test of Logical 
Thinking (TOLT) to assess five modes of formal reasoning, including controlled variables, proportional reasoning, 
combined reasoning, probabilistic reasoning, and correlational reasoning. The results demonstrated the validity 
and reliability of these measures. Bayram and Comek (2009) conducted activities that integrated network-assisted 
chemistry teaching and discovered a significant correlation between students’ chemistry scores and their scientific 
learning attitude and logical thinking abilities. Popular science e-books have been used as effective tools to explain 
abstract scientific concepts and integrate logical thinking with knowledge content. It has been established that 
engaging in popular science reading enhances comprehension and positively impacts students’ logical thinking 
abilities (KILI & SAGLAM, 2015; Nigro, 2022; Riyanti & Karyanto, 2019). 

The OKAR strategy, which was derived from SQ3R and modified by Pauk (1964), has been found to be an ef- 
fective tool in improving students’ reading comprehension skills and enhancing reading effectiveness, as noted by 
Wulandari and Amri (2013). The OKAR strategy comprises six steps: Overview (O), Keywords (K), Read (R1), Recite 
(R2), Relate (R3), and Review (R4) (Ewumiet al., 2013; Pratiwi, 2019; Tursiva & Ernalis). By employing the OKAR reading 
steps in teaching, students can read in an organized and efficient manner. This approach helps them easily identify 
key or main points in each paragraph, facilitating a better understanding of the text, and it also aids in retaining 
the information in their memory for an extended period (Karimaliana et al., 2020; Safitri, 2017; Wulandari, 2018). 

In recent years, numerous studies have demonstrated the effectiveness of mobile learning in educational 
settings (Ghazivakili et al., 2014; Gil-Flores et al., 2012; Hsu et al., 2013). For example, Huang et al. (2010) employed 
Mobile Personal Learning Environments (MPLS) on Personal Digital Assistants (PDA) for outdoor learning courses, 
which received high praise from students and proved to be a valuable tool for enhancing learning experiences 
outside the classroom. Inside the classroom, teachers can harness mobile platforms and software to boost student 
engagement and encourage cooperation, fostering a deeper understanding of collaborative work among students. 
This approach has become a prominent and widely utilized teaching method worldwide (Storkova & Kysela, 2015). 

Comparative research by Lin (2014), which pitted mobile devices against personal computers, found that mobile 
devices outperformed the PC group in terms of student participation, reading times, and reading effectiveness, 
particularly in the context of extensive reading. Similarly,Lee (2015) employed the C-QRAC reading strategy for online 
cooperative learning with scientific texts. The use of online reading strategies led to significant improvements in 
students’ scientific text reading proficiency and critical thinking skills, underscoring the benefits of employing such 
strategies in educational settings. Digital assistance systems, incorporating graphics, color changes, and game play, 
boost student learning motivation and efficiency compared to traditional methods. Mobile devices show superior 
impact over personal computers (Su & Cheng, 2013; Yang & Chang, 2013). 
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Research Aim and Research Questions 


This study aims to examine the impact of using the OK4R mobile platform for students’ science reading abil- 
ity and logical thinking ability. The specific research goals are as follows: Examine how the use of OK4R reading 
steps on mobile platforms affects students’ science reading ability. This involves assessing whether this approach 
enhances students’ comprehension of scientific content. Explore the impact of using the OK4R reading steps on 
the mobile platform on students’ logical thinking ability. This objective seeks to determine whether employing 
the OKAR strategy on mobile devices enhances students’ logical thinking skills. Compare the differences in the 
improvement of popular science reading comprehension and logical thinking abilities among vocational students 
of different genders using the OK4R platform on mobile devices versus traditional paper-based OK4R for popular 
science reading. This aspect of the study aims to discern any gender-related variations in the effectiveness of the 
OKAR strategy on mobile platforms. Investigate how gender, basic abilities, and other factors influence the effec- 
tiveness of vocational students’ use of the OK4R reading steps for popular science reading on the mobile platform. 
This objective will assess whether specific factors, such as gender or basic abilities, impact how students benefit 
from using the OK4R strategy on mobile devices for science reading. 


Research Methodology 
Research Design 


This study utilized experimental research methods to examine the impact of the “OK4R mobile platform” 
on students’ science reading ability and logical thinking ability. A control and experimental group design was 
employed to address the research questions and assess the effects of extended reality education on student 
learning responses and performance. In experimental research, researchers collect data to determine the effects 
of specific treatments. In this context, the study likely involved providing a particular treatment, which is the use 
of the “OK4R mobile platform,’ to an experimental group of participants, with the aim of assessing the benefits of 
this treatment on their science reading and logical thinking abilities (Creswell & Creswell, 2017). The research was 
carried out during the period between March and May in 2023. 


Participants 


This study involved 90 first-year students from a vocational school in Taiwan, comprising two classes with 45 
students in each class. The participants were selected through a convenient sampling method and comprised 33 
males and 57 females. The participants indicated that they used tablets and smartphones before participating in 
this research. 


Procedures 


In a quasi-experimental design, the study participants were divided into two groups, with 45 students in the 
experimental group and 45 students in the control group. Both groups of students were taught by the same lec- 
turer. The study focused ona Chinese liberal arts reading course and selected popular science articles for teaching. 
Over the course of six weeks, students participated in a popular science reading course, covering one section each 
week, with each section lasting fifty minutes. 

Every week, a text was uploaded to the platform for the students. In the first week, the OK4R mobile platform 
and the six steps of reading were introduced. Students in the experimental group used the platform during class 
to read the article, which took about 20 minutes. They also used the OK4R reading steps to summarize the key 
points, ask questions, and engage in critical thinking. In contrast, the control group used paper books to carry out 
the OKAR reading steps in the classroom. Following their reading sessions, both groups were given a paper-based 
reading test, which included multiple-choice questions and short-answer questions. 


Tools 


The tools used in this study include “The OK4R Mobile Platform’, “The Popular Science Reading Comprehen- 
sion Test” and “Logical Thinking Ability Test.’ 
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OK4R Mobile Platform 


The “OKA4R mobile platform’ is a specialized mobile application designed and developed based on the OK4R 
reading strategy. This platform enables students to effectively implement the OKAR strategy when reading popu- 
lar science articles. The platform has been created using APP Inventor2. To access it, students scan a QR code to 
download the APP program, and upon opening it, they enter their student seat number to log in to the system. 
The platform's interface is primarily divided into two main parts: Steps and Operation Instructions: In this section, 
students can find guidance and instructions for each of the six steps of the OK4R reading strategy. This part serves 
as a reference to help students understand and follow the OK4R approach. Reading Interface: The second part of 
the platform is dedicated to the reading process itself. Once students enter this section, they can use touch-screen 
interactions to navigate through the six steps of OK4R in sequence while reading the assigned popular science 
article. Upon completing these steps, students can proceed to take a paper test, likely aimed at assessing their 
comprehension and application of the strategy. 

Additionally, the platform offers tools and storage functions that allow students to upload their personal 
reading history to the cloud. This digitizes the learning process, providing a record of their reading activities and 
progress. Overall, the OK4R mobile platform is a digital tool designed to enhance the application and effectiveness 
of the OK4R reading strategy for popular science articles. The platform interface is shown in Figure1. 


Figure 1 
OK4R Mobile Platform 
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Popular Science Reading Comprehension Test 


In this study, the reading comprehension ability test used articles with scientific topics from the Taiwan PISA 
reading literacy sample test questions. The pre-test featured an article titled “Bat,”and the post-test used an article 
titled “Little Black Mosquito,’ which was modeled after the PISA test questions. Each question within the articles 
was scored according to specific scoring rules. 

Here's a breakdown of the pre-test and post-test: 


Pre-test (“Bat”):”Message from this Article’: 3 questions, including 2 questions on retrieval and retrieval 
and 1 question on integration and interpretation. “External Knowledge”: 3 questions for reflection and 
evaluation. A total of 4 multiple-choice questions and 2 short-answer questions. 
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Post-test (“Little Black Mosquito”):”Message of this Article”: 3 questions, comprising 2 questions on re- 
trieval and retrieval and 1 question on integration and interpretation. “External Knowledge”: 3 questions 
for reflection and evaluation. A total of 4 multiple-choice questions and 2 short-answer questions. 


To ensure the quality and reliability of the test questions, expert validation was conducted. Three Chinese teach- 
ers from junior high school and vocational school assessed the test questions. They evaluated whether the selected 
texts’ nature, length, and the content of the developed questions aligned with the definition of popular science 
reading and matched the information in the articles and the external knowledge measurement objectives of PISA. 

Moreover, the difficulty level of the post-test questions was compared with that of the pre-test questions by 
students from Class 3 of the Applied English Department, who shared a similar background with the experimental 
students. The accuracy rate for the post-test was found to be 0.61, which was comparable to the accuracy rate of 
0.55 in the pre-test “Bat” among the experimental students. This suggests that the difficulty level of the two tests 
was similar. 


Logical Thinking Ability Test 


This study employed three-paragraph argument questions, which have been used in previous research 
(Markovits & Nantel, 1989; Toplak et al., 2014), to evaluate their impact on improving logical thinking ability. 
Each question was designed to present a conflict between logical validity and conclusion credibility, including 
scenarios with believable conclusions but invalid inferences, as well as unconvincing conclusions but valid logical 
inferences. These questions encompassed 3 questions for each of the four types of logical structures, resulting in 
a total of 12 questions. The reliability of the items was assessed and found to have good half-half reliability (0.51) 
and internal consistency (.64). Since the research was not originally designed for high school vocational students, 
the study conducted pre- and post-test replication reliability with a group of 41 students, consisting of 19 girls and 
22 boys, from the first-grade elite class of the high school where they teach. The reliability of this assessment was 
determined to be 0.773 (p < 0.007), indicating strong internal consistency. Additionally, the accuracy rates for the 
pre-test and post-test were 54% and 51%, respectively, demonstrating the test's ability to discriminate between 
different levels of logical thinking ability. 


Data Analysis 


This study used SPSS statistical software for quantitative statistical analysis. Background variables such as 
subjects’ gender, Chinese literacy scores and science scores were collected. The independent sample t-test was 
used to compare the improvement effect of popular science reading comprehension and logical thinking ability 
between the experimental group and the control group after the OK4R reading steps. Paired samples T-test tests 
the improvement effect of pre-and post-test in different groups and genders. A two-factor multivariate analysis 
was conducted to compare the differences between higher vocational students of different genders in using the 
OKAR platform and paper OKA4R for popular science reading. Pearson product-difference correlation analysis was 
used to explore the relationship between gender, Chinese literacy scores, science scores, popular science reading 
comprehension ability and logical thinking ability. Finally, a multiple regression analysis was conducted to explore 
the prediction of gender, Chinese literacy scores, science scores, platform use, and their interaction on science 
reading comprehension and logical thinking abilities. 


Research Results 
Independent Samples T-test on the Improvement Scores of the Experimental Group and the Control Group 
In Table 1, the results show a significant difference between the experimental group and the control group 


in terms of improvement scores in popular science reading. However, there is no significant difference in improve- 
ment scores for logical thinking ability between the two groups. 
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Table 1 


Comparison Table of Progress Scores Between the Experimental Group and the Control Group 


Item The experimental group The control group t p 
Progress in popular science reading .89(1.335) .22(1.565) -2.174 .032* 
Logical thinking progress 1.40(2.580) 1.22(2.131) -.356 122 


*p<.05; **p<.01; **p<.001 


Differences in the Improvement of Popular Science Reading Ability Between Different Genders and Different Groups 


Based on the data presented in Table 2, girls outperformed boys in popular science reading among a total of 
90 experimental students. 


Table 2 
Two-Factor Variance Analysis of Gender and Group on Popular Science Reading 


Source of variation SS df us F p 
Gender 5.549 1 5.549 4.555 .036* 
Group 4.007 1 1.007 3.289 073 
Gender * Group 1.975 1 1.975 1.621 .206 
Total 1471.000 90 


*p<.05; **p<.01; ***p<.001 


Differences in the Improvement of Logical Thinking Ability Between Different Genders and Different Groups 


The improvement of their logical thinking according to different genders and different groups is shown in Table 
3. The analysis is as follows: in terms of logical thinking ability, girls perform better than boys. The experimental 


group has better logic improvement than the control group. The OK4R mobile platform has a better impact on 
logic. The improvement of thinking is effective. 


Table 3 
Two-Factor Variation Analysis of Gender and Group on Logical Thinking 


Source of Variation SS df Ms F p 
Gender 73.192 1 73.192 1.517 .002** 
Group 37.034 1 37.034 5.321 .023* 
Gender * Group 5.118 1 5.118 135 394 
Total 3079.000 90 


*p<.05; **p<.01; ***p<.001 
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Correlation Between Variables 


Pearson product-moment correlation analysis was employed to examine the relationships between various 
variables, including gender, Chinese literacy scores, science scores, popular science reading comprehension abilities, 
logical thinking abilities, and others. The findings are outlined in Table 4. There were 14 positive correlations anong 
the variables, indicating that these factors tend to influence one another. Notably, gender was found to affect popu- 
lar science reading comprehension before the test, logical thinking pre-test, and logical thinking post-test scores. 
High scores in Chinese literacy were positively associated with high scores in popular science reading pre-tests 
and popular science reading tests, suggesting a relationship between linguistic skills and science comprehension. 
Similarly, high scores in science were positively correlated with high scores in popular science reading pre-tests 
and logical thinking post-tests, highlighting a connection between science proficiency and cognitive abilities. 
This variable exhibited positive correlations with gender, Chinese literacy scores, and science scores, indicating 
that these factors influence performance in the popular science reading pre-tests. High scores in logical thinking 
pre-tests were associated with improvements in both popular science reading post-tests and logical thinking post- 
tests, suggesting a link between critical thinking skills and comprehension. Those who scored high in the logical 
thinking post-test tended to have higher logical thinking progress scores, highlighting the importance of ongoing 
development in this skill. Notably, there were two significant negative correlations: between the popular science 
reading pre-test and progress scores, and between the logical thinking pre-test and progress scores. This suggests 
that the use of OK4R reading techniques can be particularly beneficial for individuals who initially perform poorly 
in popular science reading and logical thinking. 


Table 4 
Pearson Product-Difference Correlation Analysis of Nine Variables 


Gender Chinese science Popular Logical Popular Logical Progress Logical 
literacy science thinking science thinking inpopular thinking 

reading pre-test reading post-test science progress 
pre-test post-test reading 

Gender - 

Chinese Literacy 044 

Science -.107 .307** - 

Popular Science 7 rr - : 

Reading Pre-test a ee ae 

pedicel Tanking 235" 103 193 196 

Pre-test 

Popular Science rt 

Reading Post-test 201 AN 181 .089 210 - 

eps Tne 296"* 156 237" 068 624" 351" é 

Post-test 

ogres in auer - sags -143 -.080 -.637** 024 M2 204" ; 

Science Reading 

Euical hipaa 091 078 OTT -137 -.369** 184 496"* 239" : 


Progress 
*p<.05; **p<.01; ***p<.001 


Predictive Analysis Between Variables in the Experimental Group 


In this predictive analysis, the goal was to understand how three variables (gender, Chinese literacy scores, 
and science scores) relate to the experimental group's improvement in science reading comprehension scores 
when using the OK4R mobile platform for popular science reading. Multiple regression analysis was employed 
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to uncover the predictors of platform effectiveness, and the results are summarized in Table 5. Students in the 
experimental group who used the OK4R mobile platform experienced significant improvements in both popular 
science reading comprehension and logical thinking abilities. The analysis revealed a negative predictive relation- 
ship between the students’ scores in science and their progress in popular science reading comprehension. This 
means that students who initially scored lower in science exams experienced more significant improvements in 
their reading comprehension when using the OK4R platform for popular science reading. 


Table 5 
Multiple Regression Analysis of Three Variables on the Experimental Group's Improvement in Science Reading Comprehension 
and Logical Thinking 


b SE B t p 

Popular Science Reading 

Constant 1.592 1.007 1.581 122 

Gender 391 408 146 .959 344 

Chinese Literacy .060 294 032 .203 840 

Science -.748 335 -.346 -2.235 .031* 
Logical Thinking 

Constant -1.940 1.875 -1.035 307 

Gender 911 .759 .189 1.200 238 

Chinese Literacy 1345 547 102 631 532 

Science 429 623 11 689 495 


*p<.05; **p<.0l; ** p<.001 


Predictive Analysis Among Variables in the Control Group 


In this predictive analysis, the focus was on understanding how three variables (gender, Chinese literacy 
scores, and science scores) relate to the control group’s science reading comprehension improvement scores when 
using the paper-based OK4R approach for popular science reading. Multiple regression analysis was performed to 
identify the predictors of the paper-based OK4R's effectiveness, and the results are presented in Table 6. Students 
in the control group, even when using the paper-based OK4R method for popular science reading, experienced 
an improvement in their logical thinking abilities. The analysis did not reveal any of the three variables (gender, 
Chinese literacy scores, and science scores) as predictors of the progress in science reading comprehension or 
logical thinking improvement for the control group. 


Table 6 
Multiple Regression Analysis of Three Variables on the Improvement in Science Reading Comprehension and Logical Thinking 
in the Control Group 


b SE Bp t p 
Popular Science Reading 
Constant 1.094 1.312 834 409 
Gender -.116 516 -.035 -.224 824 
Chinese Literacy -.572 .340 -.273 -1.685 100 
Science 371 .368 167 1.008 319 
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b SE p t p 
Logical Thinking 
Constant 841 1.853 454 652 
Gender 147 128 .032 .202 841 
Chinese Literacy -.022 480 -.008 -.046 963 
Science 101 520 .033 .194 847 


*p<.05; **p<.01; ***p<.001 


Predictive Analysis of the Five Variables of All Students on the Improvement Scores of 
Popular Science Reading Comprehension 


The results of a multiple regression analysis were conducted to investigate the impact of various factors on 
the improvement of popular science reading comprehension in both an experimental group and a control group 
of students. In the experimental group, it was found that students’ science scores negatively predicted their prog- 
ress in popular science reading. However, this effect was not observed in the control group. To further explore the 
impact of different variables on students’ progress in popular science reading comprehension, an interaction term 
between group (experimental or control) and science scores was added as a predictor. Additionally, an interaction 
term between Chinese literacy scores, science scores, group, and science scores was included in the analysis. The 
results are presented in Table 7. Among all students, it was found that the use of the OK4R mobile platform was an 
effective predictor of improvement in science reading comprehension. In the experimental group specifically, it 
was observed that students’ natural science performance negatively predicted their progress in science reading. 


Table 7 
Multiple Regression Analysis of the Five Variables of All Students on the Progress Scores of Popular Science Reading Compre- 
hension 


b SE B t p 
Popular Science Reading 
Constant 1.438 .650 2.211 .030 
Gender 171 330 .055 518 606 
Group 1.243 480 838 2.589 .011* 
Chinese literacy -.264 227 -.128 -1.162 .249 
Science -.183 252 -.081 -.726 470 
Group*Science -A74 241 -.634 -1.965 .053* 


*p<.05; *p<01; **p<.001 


Predictive Analysis of the Five Variables of All Students on Their Logical Thinking Progress Scores 


Ina multiple regression analysis, several predictor variables were used to predict the improvement in logical 
thinking ability among all students. These predictor variables included gender, whether the platform was used 
or not, Chinese literacy scores, science scores, and the interaction between group and science scores. However, 
the results, as presented in Table 8, indicate that none of the predictor variables reached statistical significance in 
predicting the improvement in logical thinking ability. This means that gender, platform usage, Chinese literacy 
scores, science scores, and the interaction between group and science scores were not found to be significant 
predictors of changes in logical thinking ability among the students. 
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Table 8 
Multiple Regression Analysis of the Five Variables of All Students on Their Progress in Logical Thinking 


b SE B t p 
Logical Thinking 
Constant -.113 1.029 -.110 913 
Gender 559 521 120 1.071 .287 
Group -.400 .760 -.179 -.526 600 
Chinese Literacy 164 359 .053 455 650 
Science .287 .398 .084 720 AT4 
Group*Science .199 381 ATT 522 603 


*p<.05; **p<.01; ***p<.001 


Discussion 


The aim of this study is to assess the effectiveness of the OK4R reading strategy when delivered through a 
mobile platform app in improving students’ popular science reading comprehension and enhancing their logi- 
cal thinking abilities. Additionally, the research endeavors to examine how various factors, including gender and 
students’ fundamental cognitive abilities, may impact the results and outcomes of this educational approach. To 
facilitate this study, a teaching tool was created, which is grounded in the OK4R reading strategy, and was imple- 
mented through a mobile platform app. The research findings indicate significant differences when comparing 
the pre- and post-test scores of popular science reading comprehension between the experimental group (using 
the OK4R mobile platform) and the control group (employing traditional paper-based OK4R reading steps). Spe- 
cifically, the experimental group exhibited notably higher improvement scores compared to the control group. 

Furthermore, in terms of correlations, the study revealed that progress scores in popular science reading 
comprehension were not significantly affected by gender. However, there was a negative correlation between 
these progress scores and the pre-test scores of popular science reading, suggesting that students with lower initial 
scores tended to benefit more from the intervention. Multiple regression analysis also demonstrated a significant 
relationship between the progress scores of popular science reading and the group (OK4R mobile platform or tra- 
ditional paper-based steps). This indicates that the use of the OK4R mobile platform was an effective predictor of 
improved science reading comprehension for all students. Overall, the results suggest that using the OK4R mobile 
platform for science reading can effectively enhance science reading comprehension for both male and female 
students, with the greatest benefits observed among those who initially had lower scores in science reading ability. 

The analysis of pre-test and post-test scores for logical thinking abilities between the experimental group 
(using the OKA4R reading steps) and the control group revealed a significant difference. This suggests that popular 
science reading, regardless of the platform used, effectively enhances logical thinking abilities when employing the 
OKAR reading strategy. In terms of correlations, there were no significant differences in logical thinking progress 
scores between genders. However, a positive correlation was observed between logical thinking progress scores 
and popular science reading progress scores, indicating that as students’ popular science reading improved, their 
logical thinking abilities also improved. Conversely, there was a negative correlation between logical thinking 
progress scores and their respective pre-test achievements, suggesting that students with lower initial logical think- 
ing scores experienced more significant improvements. Multiple regression analysis did not identify any predictor 
variables for the improvement in logical thinking. However, it was concluded that utilizing the OK4R reading steps 
for popular science reading effectively enhances logical thinking abilities for both male and female students. Ad- 
ditionally, students who demonstrated higher progress in science reading also showed greater improvements in 
logical thinking, with the most significant improvements observed among those who had lower pre-test scores 
in logical thinking ability. 

However, from the perspective of gender learning outcomes, there is no significant difference in the two ba- 
sic abilities of the subjects of this study in the Chinese Language and Literature Examination and Natural Science 
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Examination scores. However, the performance of women in the pre-test of popular science reading and the pre- 
test of logical thinking is significantly better than that of men. When using the OK4R mobile platform for popular 
science reading, both boys and girls can effectively improve their science reading comprehension and logical 
thinking abilities. Gender cannot significantly predict the improvement of science reading comprehension and 
logical thinking abilities. This is consistent with Sanchez and Wiley (2010) in the literature. Research by “Men have 
traditionally performed better than women in visual-spatial abilities and overall science learning’, and research by 
Lee (2015) “Gender differences have an impact on scientific reading and learning outcomes, but teaching strate- 
gies or methods can also be used, leveling out gender differences in the effectiveness of popular science reading. 

In the study involving a total of 90 experimental students, the distribution of students across groups and 
gender is as follows: In the experimental group, there were 19 boys and 26 girls, while in the control group, there 
were 14 boys and 31 girls, resulting in a total of 33 boys and 57 girls across both groups. The analysis of differences 
in scores for science reading ability and logical thinking ability after popular science reading, based on gender 
and platform usage, was conducted using a two-factor multivariate analysis. The results revealed significant dif- 
ferences between genders in both popular science reading ability and logical thinking ability. However, there was 
no significant interaction between gender and group, indicating that, overall, females outperformed males in both 
popular science reading and logical thinking abilities. Furthermore, a significant difference was observed in logical 
thinking ability between the two groups, with the experimental group, which utilized the OK4R mobile platform, 
demonstrating superior performance in logical thinking abilities compared to the control group. In summary, the 
findings suggest that there are notable gender differences in popular science reading and logical thinking abilities, 
with females performing better than males in both areas. Additionally, the use of the OK4R mobile platform in the 
experimental group led to enhanced logical thinking abilities compared to the control group. 

The analysis of predictor variables, including students’ gender, group (experimental or control), Chinese lit- 
eracy scores, and science scores, revealed significant findings regarding popular science reading comprehension 
progress scores and logical thinking progress scores. 

The experimental group using the OK4R mobile platform was found to be a significant predictor of popular 
science reading comprehension improvement, suggesting that this platform positively impacts students’ progress 
in popular science reading. 

Science scores in the experimental group had a negative predictive effect on popular science reading com- 
prehension improvement, indicating that higher science scores were associated with less progress in popular 
science reading. Chinese literacy scores did not significantly affect popular science reading comprehension or 
logical thinking abilities. 

The interaction between group and science scores was a significant and negative predictor of science reading 
improvement, suggesting that the design of the OK4R mobile platform was particularly beneficial for low-achieving 
students in improving their science reading abilities. There was no significant correlation between gender and the 
two basic scores (Chinese literacy and science), but there was a positive correlation between pre-test scores for 
popular science reading and logical thinking. After experimental teaching, there was a significant improvement 
in women's logical thinking post-test scores. However, there was no significant correlation between the overall 
progress in science reading and logical thinking progress scores. 

The study revealed several important findings: Female students performed better in both the science read- 
ing pre-test and logical thinking pre-test compared to their male counterparts. Female students also exhibited 
greater improvement in logical thinking, as indicated by their post-test scores. Interestingly, students with lower 
scores in science achieved better results in popular science reading when using the OK4R platform. The greater 
the progress made, the more significant the benefits observed. Overall, the study demonstrated that the OK4R 
platform is particularly effective in improving popular science reading comprehension for students who have lower 
scores in natural science. This suggests that the platform can be especially beneficial for those who may initially 
struggle with science-related content. These findings underscore the potential of the OK4R platform to address 
disparities in science reading comprehension and benefit students, especially those who face challenges related 
to their performance in science scores. Therefore, echoing the research of Karimaliana et al. (2020) and Lee (2015) 
that employed similar strategies for enhancing reading and critical thinking skills in the context of scientific texts. 


Conclusions and Implications 


The study indicates that the application of OK4R reading strategies is effective in improving students’ science 
reading comprehension and logical thinking abilities. 
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The study also highlights the following key points: Gender alone does not seem to predict differences in 
academic performance, as there were no significant gender disparities in Chinese literacy and science scores. 
The study found that women outperformed men in the pre-tests of popular science reading and logical thinking, 
emphasizing that gender is not the sole determinant of improvements in these abilities. Both male and female 
students benefitted from the OK4R reading steps, leading to improvements in science reading comprehension and 
logical thinking. This suggests that these strategies are equally effective for both genders and that gender is not 
a significant predictor of progress in these areas. While there were improvements in both genders in pre-test and 
post-test scores, the gender difference in the post-test of popular science reading was only marginally significant. 
This raises questions about the extent to which OK4R reading strategies can reduce gender disparities in popular 
science reading, with the study hinting that the effect might not be substantial. 

The study provides valuable insights into the effectiveness of OK4R reading strategies in enhancing science 
reading comprehension and logical thinking abilities for students of all genders. However, it also suggests that 
further research is needed to determine the full impact of these strategies on reducing gender differences in 
popular science reading. 


Recommendations 


The study hints at the need for further research with larger participant samples and extended training periods 
to assess the full impact of OK4R reading strategies in reducing gender disparities in popular science reading. 
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Abstract. This study examined the effects 
of students’ argumentation standpoints 
on their argumentation learning in the 
context of socio-scientific issues (SSIs). 

To that end, four kinds of argumentation 
standpoints were defined: affirmative 
standpoints, oppositional standpoints, 
multiple standpoints, and non-standpoints. 
These four kinds of standpoints allow for 
six possible combinations of any two of 
the different kinds of standpoints, which 
enabled us to conduct six kinds of 2-team 
format debates. The resulting differences 
of students’ four types of arguments (i.e., 
claims, warrants, rebuttals, and qualifiers) 
generated in six types of debates were 
examined. This study invited 208 10th- 
grade students to participate in a quasi- 
experimental research design. The results 
showed the affirmative group students 
demonstrated superior performance in 
terms of claims and warrants, and the 
oppositional group students had the 
largest number of rebuttals. The students 
in the debate with combinations of 
affirmative and oppositional groups 
exhibited the best performance regarding 
the generation of claims, warrants, 

and rebuttals. Based on the results, the 
present study concluded the standpoint 
had significant effects on the students’ 
argumentation learning, which suggests 
that teachers can investigate students’ 
standpoint on the learning topic of SSI and 
their prior knowledge about the standpoint 
before teaching. 
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Introduction 


The use of socio-scientific issues (SSIs) in teaching science has been 
reported to be effective because such lessons provide learners with oppor- 
tunities to engage in informal reasoning (Dawson & Venville, 2009; Zeidler 
et al., 2019), scientific explanation (Eggert et al., 2016; Gutierez, 2015), and 
argumentation (Cho & Jonassen, 2002; Grace & Ratcliffe, 2002). Ideally, the 
main goal for students in scientific argumentation activity is to engage in 
critical analyses of others’ arguments. SSI debates constitute a typical form 
of scientific argumentation activity in which students are expected to assess 
the advantages and disadvantages of each position regarding the given is- 
sue (Zeidler et al., 2019; Ke et al., 2020). Previous researchers have reported 
that most middle school students are capable of articulating at least one 
perspective or standpoint/position in the context of an SSI activity (Lin, 2023; 
Anwar & Ali, 2020; Tekbiyik, 2015). Based on our own prior investigations, 
such student argumentation standpoints can be further categorized into four 
kinds of standpoints according to their stances in relation to the proposi- 
tion under consideration: affirmative standpoints, oppositional standpoints, 
multiple standpoints, and non-standpoints. Taking the commonly discussed 
SSI “the nuclear power plant” as an example, a student who holds the af- 
firmative standpoint would say “Yes, it would be no problem to construct a 
nuclear power plant in our country”; a student with an oppositional stand- 
point would argue that “no, building a nuclear power plant would be a bad 
decision”; and a student who holds multiple standpoints would consider a 
number of standpoints before making his or her argument. Such students 
may also say something like, “It depends; both decisions have their respec- 
tive pros and cons, and so this issue requires further investigation”. A student 
taking a non-standpoint usually has no clear preference regarding the SSI 
and may state that “both [the affirmative and oppositional standpoints] are 
acceptable” (or that “both are unacceptable”). 

Even though the mere expression of standpoints may not bea problem 
for most beginners or high school students, researchers have found that 
rigorous instructions and scaffoldings are still required in order to improve 
their argumentation abilities (Lin, 2023; Ekborg, 2008; Lee & Grace, 2010; 
Sadler, 2004; Zeidler et al., 2002). For example, Ratcliffe (1997) reported 
that it was rare for 15-year-old students to use scientific information in their 
reasoning and decision making during the discussion of SSIs. As she put it, 
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“Scientific concepts may not appear as the most important factors in making decisions about socio-scientific 
issues (p. 179)". Walker and Zeidler (2007) developed a web-based inquiry-focused curricular unit to scaffold 
high school students’ argumentation. They indicated that they still tend to back their assertions with personal 
opinions instead of evidence. Such a tendency “ultimately [leads] into numerous instances of fallacious reasoning 
and personal attacks (p. 1403)”. To sum up, high school students may be lacking in argumentation and science 
content knowledge, and this lack of knowledge may cause them to rely exclusively on their personal opinions 
and only a limited amount of relevant information in arguing about SSls (Lin, 2022; Hong & Chang, 2004; Klopp 
& Stark, 2022).To provide scaffolds for students’ argumentation knowledge, a number of research used Toulmin’s 
argumentation theory (Toulmin, 1958) as a fundamental data analysis foundation (Erduran et al., 2004; Weinberger 
et al., 2010). Toulmin explains that a sound argument will typically include six elements: data, claims, backings, 
warrants, rebuttals, and qualifiers (Toulmin, 1958). Researchers extended the Toulmin’s theory and explained a 
number of aspects of knowledge and strategies required to construct a sound argument, including what counts 
as evidence, why counter-argument should be constructed, how to back a claim with scientific evidence, and so 
on (Erduran et al., 2004). 

A number of studies have investigated how students’ personal value and prior knowledge about the SSI 
influence their standpoints and decision making (Casas-Quiroga & Crujeiras-Pérez, 2020; Dauer et al., 2017; Her- 
renkohl & Guerra, 1998; Sadler, 2004). For example, Dauer and her colleague (2017) explored university students’ 
reasoning process regarding SSls. They explained that an individual’s decision making was influenced by personal 
value orientations. However, in the SSI debate, the predictions from individuals’ personal factors to decision mak- 
ing should be modified, or even reversed, because most of students have already made a stance before attending 
a debate. In our perspective, a standpoint a student made before debate would not only reflect his or her prior 
value, prior preference, and prior knowledge but would also affect the subsequent development of argumenta- 
tion abilities. The current study explored how students’ standpoints influence their science argumentation in the 
context of SSI. To that end, four kinds of argumentation standpoints were defined, which enabled us, in turn, to 
sort the participating students into four corresponding groups: the affirmative group (A group), the oppositional 
group (O group), the multiple-standpoint group (M group), and the non-standpoint group (N group). From the four 
groups, six distinct types of student debates can emerge. That is, debates between the A and O groups, debates 
between the A and M groups, debates between the A and N groups, debates between the O and M groups, debates 
between the O and N groups, and debates between the Mand N groups. More specifically, by identifying the four 
kinds of standpoints and the six kinds of student debates, we can explore which type of student standpoint and 
which type of debate would benefit students’ argumentation learning most. The research questions: (1) which 
kind of student argumentation standpoint regarding an SSI benefits students most in terms of the development 
and improvement of the four types of arguments (i.e., claims, warrants, rebuttals, and qualifiers) in the context of 
a within-group interview activity? (2) Which type of student debate yields the best argumentation and the most 
improvement in terms of the four types of arguments? 


Research Methodology 
Procedure 


The current study utilized a quasi-experimental design. Before any argumentation instructions were pre- 
sented to the participants, a number of SSIs were introduced to them in order to investigate their standpoints 
with respect to the SSIs. Two environmental SSls, i.e., the “Su-Hua highway project” and “chemical cosmetics” 
SSIs, were then selected as they led to approximately equal distributions among the four kinds of standpoints. 
The experimental educational research on two SSI units lasted for eight weeks. Students were assigned to four 
groups based on their standpoints, and each group was further subdivided into three subgroups to engage in 
different argumentation activities; for example, students with A standpoint were divided into three subgroups 
participating in argumentation/debate activities with groups O, M, and N. To ensure that each subgroup had at 
least five students for better engagement in the activities, this study opted for a single experimental group design. 

Before conducting the first experimental SSI activity, instructors spent two hours introducing the students 
to basic knowledge regarding argumentation, grouping them according to their standpoints on the SSI, explain- 
ing to them the basic strategies involved in collaborating with their peers to search for relevant information. 
The size of the student discussion group was five to six members. After the introductory lesson, all the students 
were given two to three hours to investigate related information on the internet, in the school library, and in 
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their lab if necessary. They then participated in four 30- to 40-minutes long argumentation activities in order to 
learn about a given SSI. The first activity was a reflective judgment interview (RJI), which was conducted at the 
beginning of the autumn semester for the school teachers to choose scientific argumentation as the main topic 
for their professional development. The other follow-up three argumentation activities were two-team format 
debates in which a group of students with one kind of standpoint debated with, respectively, three groups of 
students with a different type of standpoint. The RJI is a semi-structured interview designed to elicit reliable data 
about an individual's (or individuals’) fundamental beliefs about a particular concept/topic (King & Kitchener, 
1994). Different from the RJI, the two-team format debates emphasized the dialogue argumentation between 
the different student groups. The teachers maintained a neutral stance when asking probe questions during the 
RJI and debate activities. 


Participants 


The participants of the study included two science teachers, as well as four classes comprising a total of 
208 students who were in the 10th grade. Among these students, there were 198 female students and 10 male 
students. They were students from a regular high school located in the suburban area of central-southern Taiwan. 
The socioeconomic status of the community where the school is situated tends to be middle-class. The majority of 
students in the school specialize in nursing and biology. The school assigned all the students randomly to different 
classes. Thus, a relatively even distribution of students in terms of different abilities and levels of prior knowledge 
was assumed. The ANOVA results for the students’ final science examination in the previous semester (pre-test) 
showed no significant differences in the levels of prior science knowledge among the A group (N = 48), O group 
(N = 58), M group (N = 57), and N group (N = 45) (f= .021, p > .05). The two teachers selected for participation in 
the present study were both Ph.D. candidates. They both majored in science education and had both obtained 
five years of experience in teaching science in the school. 


The SSIs and Questions for Argumentation 


The first SSI topic selected was the Su-Hua highway project SSI. The project was first proposed in 2011 and 
has since caused much discussion and argumentation among both the relevant governmental authorities and 
the citizens of this country. The probe questions about this SSI used by the two teachers in order to engage the 
students in argumentation were “Should we agree or disagree with efforts to undertake the Su-Hua highway 
project? What are your opinions and suggestions?” The second one concerned the use of cosmetics. The probe 
questions were “Do you believe that products of cosmetic are indispensable in a woman's daily life? Share your 
arguments and perspectives to substantiate your opinion”. 


Data Collection and Analysis 


Quantitative analysis. One-factor repeated measure ANOVA and MANOVA analysis were used to explore the 
performance of students’argumentation during the two SSI activities, including their progressions and differences 
among the four groups in terms of using the four types of arguments. 

Qualitative analysis. Based on the coding framework developed by previous studies (Erduran et al., 2004; 
Hogan & Maglienti, 2001), the Toulmin’s theory was applied to develop an analytical framework for coding student 
arguments into types and categories. Each student argument generated during RJIs and debates was classified into 
one of the four types of arguments: claim, warrant, rebuttal, and qualifier. Furthermore, each student argument 
was Classified into one of two levels of quality according to three principles: a) “Is the given argument rational, 
logical, and coherent?”; b) “Does the given argument state the source of authority and provide a relevant expla- 
nation?”; and c) “Is the given argument based on theoretical or empirical evidence?” If an argument met at least 
one of the three criteria above, it was deemed a level 2 argument. If an argument did not meet any of the three 
criteria, it was deemed a level 1 argument. In the quantity analysis, Level 1 arguments were assigned a code of 1, 
while Level 2 arguments were assigned a code of 2. All of the students’ arguments generated during the RJIs and 
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debates were videotaped and transcribed. The resulting transcripts were then read line-by-line by the first author 
and the two teachers. Any parts of the conversation that were not related to the task or topic were removed. Any 
inconsistencies were discussed further, and the cross-coder reliability was .89. 

Analysis of students’ dialogue argumentation. The strategies of content analysis were applied to explore 
qualitative data in order to ermine learning differences among the four standpoint groups, and, furthermore, to 
explain the results and findings of the quantitative analyses. We applied the coding framework constructed in our 
previous studies to analyze the students’ performance in argumentation (Authors, 2022, 2023). 


Research Results 
Student Argumentation in Interview Activity 


Figure 1 shows that all four groups of students produced more level 1 arguments during the second SSI in- 
terview than during the first one. A one-factor repeated measure ANOVA reveals that all four student groups had 
a significant improvement in their frequency of level 1 arguments in the second SSI interview (A group, f,. = 28.91, 
p< .001; O group, f= 39.26, p < .001; M group, f,, =30.55, p < .001; N group, f= 6.05, p < .05). Among the four 
groups, the O group exhibited the largest improvement. A simple one-factor ANOVA was then conducted to 
examine the effects of the different kinds of standpoints on the four level 1 arguments. The results indicate that 
there was no significant difference among the four student groups during the first SSI interview. However, a dif- 
ferent result was found in the analysis of the second SSI interview. The data not being homogenous (f,, = 7.58, p 
< .01), the Welch test was used, and it shows that the adjusted value was fine (Welch = 10.31, p < .01). The Games- 
Howell post hoc further shows that the N group students produced significantly fewer level 1 arguments than the 


A group students (MD = .89, p < .01), O (MD =.99, p< .01), and M (MD = ..98, p < .01) groups. 


Figure 1 
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Figure 2 
Mean Frequencies of Level 2 Arguments Used by Students in Four Group Types During SSI Interviews 
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The descriptive statistics regarding level 2 arguments show that all four student groups produced more of 
the four types of level 2 arguments in discussing the second SSI topic than in discussing the first one (Figure 2). A 
one-factor repeated measure ANOVA with the SSI topic as a repeated factor was therefore conducted to investigate 
the degree of their improvements. The results show that students in the A (f= 39.24, p < .001), O (f,,=1 18.07, 
p <.001), and M (f,. = 150.94, p < .001) groups exhibited a significant improvement in the second SSI interview. 
Following that, a one-factor ANOVA was executed, with the standpoint serving as the independent variable and 
the frequency of Level 2 arguments as the dependent variable. The results indicate that the kind of standpoint 
had a statistically significant effect on the frequency of level 2 arguments produced in the first SSI interview. Since 
the homogeneity of variance of the data is invalid (f= 30.61, p < .01). The result shows the adjusted value was 
fine (Welch = 13.35, p < .01). The Games-Howell post hoc shows that the A (MD = .52, p < .05), O (MD =.67,p < .01), 
and M (MD = .60, p < .01) groups performed significantly better than the N group. Importantly, the analysis of the 
second SSI interview indicated that the effect of the kind of standpoint became greater (Welch = 142.45, p<.001).A 
Games-Howell post hoc test furthermore shows that the M group performed better than the A (MD = 2.22, p< .001), 
O (MD = 1.36, p < .001), and N groups (MD = 4.01, p < .001); the O group performed better than the A (MD = .86, 


p<.05) and N (MD = 2.65, p <.001) groups; and the A group performed better than the N group (MD = 1.79, p<.001). 


Table 1 
One-Factor ANOVA of SSI Interviews 


n Ssi1 SSI 2 SSI topics 
M sD M sD F(p) Simple main effect 
Claim 

Agroup 48 1.33 1.41 2.37 1.64 11.23" $sl2>Ssli* 

O group 58 0.86 1.26 2.09 1.40 41.25" $Sl2>SS|1*** 

M group 57 0.77 1.15 2.81 1.70 95.84*** $Sl2>Ss|1*** 

N group 45 0.36 0.48 0.31 0.46 281 

Warrant 

Agroup 48 0.67 1.03 3.10 1.58 65.12*** $sl2>Ssl1*** 

O group 58 0.71 1.21 2.62 2.08 72.69°** $sl2>Ssl1*** 
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n Ssi1 SSI 2 SSI topics 
M sD M sD F(p) Simple main effect 

M group 57 0.70 1.13 3.53 1.88 114.58*** SSI2>SSI1*** 
N group 45 0.58 0.49 0.78 0.42 6.00* $S12>SSI1* 
Rebuttal 
A group 48 0.11 0.31 0.40 0.73 6.87* $SI2>SSI1* 
O group 58 1.05 1.09 2.69 1.75 57.82*** $SI2>SSI1*** 
M group 57 0.53 0.88 1.96 1.84 53.25*** SSI2>SSI1*** 
N group 45 0.09 0.28 0.18 0.53 2.04 
Qualifier 
A group 48 0.06 0.24 0.08 0.27 1.00 
O group 58 0.17 0.65 0.38 0.95 6.01* $S12>SSI1* 
M group 57 0.58 0.99 2.19 2.10 41.68** $SI2>SSI1*** 
N group 45 0.11 0.31 0.22 0.42 2.89 


Note: ™’p < .0001, “p < .001,'p <.01 


The findings from Table 1 indicate that students in both the O and M groups demonstrated notable advance- 
ment in their ability to generate claims (the O group, i = 41.25, p < .001; the M group, fa = 91.84, p < .001) and 


rebuttals (the O group, he = 57.82, p<.001; the M group, os =53.25,p<.001). As for warrants, all four student groups 


exhibited a significant improvement during the second SSI interview (the A group, f,,, = 65.12, p <.001; the O group, 


f= 72.69, p < .001; the M group, f,, = 114.58, p < .001; the N group, f,,, = 6.00, p < .05). These results suggest that 
using claims and warrants in the SSI interviews did not seem to be a difficult task for most of the students. On the 
other hand, the results regarding the use of qualifiers were different, with the results indicating that the students 
rarely used this type of argument during the interviews. Such improvements indicate that the students gradually 
learned how to provide structure and level 2 arguments, although there were still level 1 arguments that were re- 
corded. An example of the M group students’ dialogue interactions in the RJI was provided to support our statistical 
findings. As for the A group, they had a significant improvement in their use of warrants, a slight improvement in 
their use of claims and rebuttals, and no significant improvement in their use of qualifiers. As for the N group, they 
exhibited no significant improvement with regard to any of the four types of arguments. We therefore concluded 
that, among the four student groups, the students in the O and M groups exhibited more improved performance 
in terms of their use of the four types of arguments. 


Table 2 
MANOVA Analysis of Four Arguments in the SSI Interviews 


Wilk’s A Multivariate F Univariate F Post Hoc Tests 

Claim 673 14.84*** 

Ssit 5.67" A>N group** 

ssl2 28.43*** A>N group** 
O >N group*** 
M >N group*** 
M >0 group* 

Warrant 123 11.89°** 

Ssit 162 

ssi2 25.02*** A>N group*** 
O >N group*** 
M >N group*** 
M >0 group* 
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Wilk’s A Multivariate F Univariate F Post Hoc Tests 
Rebuttal 613 18.76*** 
Ssit 18.37*** O >A group*** 
O>M group** 
O >N group*** 
M >A group* 
M >N group* 
ssl2 38.71*** O >A group*** 
O >M group* 
O >N group*** 
M >A group*** 
M >N group*** 
Qualifier 658 15.77*** 
Ssit 7.05*** M >A group*** 
M >0 group** 
M >N group** 
Ssi2 35.18*** M >A group*** 
M >0 group*** 
M >N group*** 


Note: “"p < .0001, “p <.001,"p<.01 


The results of Table 2 indicate that this kind of standpoint factor had statistically significant effects on the 
use of claims (A = .673, p < .001), warrants (A =.723, p < .001), rebuttals (A = .613, p < .001), and qualifiers (A = .658, 
p<.001). A main effect analysis was therefore performed, and it indicates that, first, the kind of standpoint had a 
significant effect on the use of claims in both the first it = 5.67, p <.01) and second if = 28.43, p < .001) SSI inter- 
views; Sidak test shows only the A group significantly outperformed the N group in first SSI interview (MD = .98, 
p <.001). The disparities among the four groups expanded further during the second SSI interview: the A group 
outperformed the N group (MD = 2.06, p < .001); the O group outperformed the N group (MD = 1.78, p < .001); and 
the M group outperformed the N group (MD = 2.50, p < .001) and the O group (MD = .72, p < .05). 

Second, in terms of the use of warrants, the kind of standpoint had no statistically significant effect during 
the first SSI interview; however, it reached a statistically significant effect during the second interview ae = 25.02, 
p< .001). Sidak test further shows that the students in the A (MD = 2.33, p < .001), O (MD = 1.84, p < .001), and M 
(MD = 2.75, p < .001) groups had significantly better scores for their use of warrants than the N group. Moreover, 
the M group also exhibited significantly better warrant usage than the O group (MD = 2.75, p < .001). Third, in 
terms of the use of rebuttals, the type of standpoint also exhibited a statistically significant impact during both 
the first C, = 18.37, p < .001) and second ie = 38.71, p < .001) SSI interviews. The homogeneity of variance in the 
data is not valid for both the first (hs 204) = 36.68, p < .001) and second ie 204) = 26.11, p < .001) SSI interviews. The 
result shows the adjusted value was fine for both the first (Welch = 121.11, p < .001) and second (Welch = 156.73, 
p <.001) SSI interviews. 

The Games-Howel for the first SSI topic shows that the O group generated a better rebuttal score than the A 
group (MD = .95, p< .001), M (MD = ..53, p< .01), and N (MD = ..96, p < .001). The disparities among the four groups 
expanded during the second SSI interview; the O group students had significantly better rebuttal scores than 
their peers in the A (MD = 2.29, p < .001), M (MD =.72, p < .05), and N (MD = 2.51, p < .001) groups. As for quali- 
fiers, the results indicate that the effect of the kind of standpoint reached statistical significance for both the first 
(Welch = 37.18, p < .001) and second SSI topics (Welch = 121.24, p < .001). The games-Howell test shows that the M 
group students had a better score in qualifiers than the A (MD=.52, p< .001),O (MD=.41, p< .01), and N (MD = 47, 
p <.01) groups in discussing the first SSI topic. For the second topic, the Games-Howell post hoc test results were 
similar to those for the first one. One difference, however, was that the mean difference and p-value for the second 
interview were higher, indicating that the M group students made greater progress than the A, O, and N group 
students in generating qualifiers. 
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Student Argumentation in Debate Activity 


A one-factor repeated measures ANOVA, with the SSI topic as the repeated factor, was performed to explore 
the extent of students’ improvements in relation to the four Level 1 arguments. The descriptive statistics show, 
in general, the frequencies with which the four level 1 arguments in the six debates during the second SSI topic 
were higher than the frequencies with which they were generated during the first one (Figure 3). Moreover, the 
students who engaged in all six types of debates made significant progress during the second SSI debate with 
regard to their frequency of level 1 arguments (AO debate, f= 7.56, p < .01; AM debate, f, = 4.38, p < .05; AN de- 


bate, fy= 68.06, p < .001; OM debate, f= 22.16, p < .001; ON debate, fy= 55.01,p< 001: MN debate, fy= 10.38, 
p <.01). A one-factor ANOVA was conducted, and the independent variable was the debate type, while the four 
types of level 1 arguments served as the dependent variables. The results show the independent variable had a 
statistically significant effect on students’ argumentation (Ff. = 3.09, p < .05). Sidak test shows only the AO debate 
had better level 1 arguments than the AN debate significantly (MD = 1.19, p < .05). 

Figure 4 shows that the frequencies of level 2 arguments generated by the students in the six debates during 
the second SSI topic were higher than that generated during the first one. Moreover, the AO and AM debates yielded 
the highest frequencies of level 2 arguments, while the AN and ON debates yielded the lowest frequencies of level 
2 arguments among the six types of debates. A one-factor repeated measure ANOVA indicates that the students 
who participated in all six types of debates made significant progress regarding the level 2 arguments during the 
second SSI debate (the AO debates, f= 78.67, p < .001; the AM debates, fy= 100.34, p < .001; the AN debates, 
fy= 28.70, p < .001; the OM debates, fy= 84.98, p < .001; the ON debates, fy= 24.70, p < .001; the MN debates, 


fy= 28.38, p < .001). 


Figure 3 
Mean Frequencies of Level 1 Arguments Used by Students in Six Types of SSI Debates 
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Figure 4 


Mean Frequencies of Level 2 Arguments Used by Students in Six Types of SSI Debates 
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A one-factor ANOVA was then applied to investigate the effect of the type of combination on the frequency 
of level 2 arguments generated in the first and second SSI debates. 

The results show that the homogeneity of variance of the data was invalid for the first SSI es =5.51,p<.01) 
and second SSI Maa = 5.83, p <.01), so the Welch test was then used. The Welch test results show that the adjusted 
values were fine for both the first (Welch = 2.57, p < .05) and second (Welch = 17.29, p < .01) SSI debates. Follow- 
up Games-Howell post hoc for the second SSI debate shows that the AO debate students produced better level 2 
arguments than the AN significantly (MD = 2.47, p < .001), ON (MD = 2.40, p < .001), and MN (MD = 1.72, p < .001) 
debate students. Moreover, the AM debate performed better than the AN (MD = 2.02, p < .001), ON (MD = 1.94, 
p <.001), and MN (MD = 1.26, p < .05) debate; the OM debate performed better than the AN (MD = 1.38, p < .01), 
and ON (MD = 1.30, p < .05) debate. Such results implied to us that the students in the AO, AM, and OM debates 
could produce more level 2 arguments during the second debate activity. 

The results in Table 3 indicate that the students participating in all six types of debates had significant improve- 
ment in terms of their scores for arguments such as rebuttals, warrants, and claims during the second SSI debate. 
It is worth noting that only three groups (the AM, OM, and MN debate students) made significant progress in their 
use of qualifier arguments. Among those three groups, we found that the MN debate students improved the most. 


Table 3 
One-Factor ANOVA Analysis of Debate Activity. 


n SSI 1 SSI 2 SSI topics 


M SD M SD F(p) Simple main effect 

Claim 

AO group 35 1.06 1.16 3.06 2.07 37.18* $sl2>Ssi1*** 

AM group 35 0.77 1.14 2.29 0.86 66.97*** $sl2>ssi1*** 

AN group 31 0.81 1.08 1.97 1.47 47.88*** $sl2>ssi1*** 

OM group 38 0.50 0.76 1.13 1.17 13.09** $sl2>ssi1** 

ON group 35 0.54 0.78 1.26 1.34 12.35** $sl2>ssi1** 

MN group 34 0.50 0.83 1.29 1.34 19.89*** SSl2>Ss|1*** 
Warrant 

AO group 35 1.34 1.16 3.31 1.89 32.39*** $sl2>ssi1*** 

AM group 35 0.97 1.15 2.94 1.68 60.08*** $sl2>ssi1*** 

AN group 31 0.52 1.03 2.19 1.33 75.25*** $sl2>ssi1*** 
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n SSI 1 SSI 2 SSI topics 
M SD M SD F(p) Simple main effect 

OM group 38 0.63 0.94 2.92 2.17 47.91*** $Sl2>SSsi1** 

ON group 35 0.49 0.78 1.69 1.57 33.20°** $Sl2>SSI1*** 

MN group 34 0.56 0.79 1.74 1.66 20.72*** $SI2>SSI1*** 
Rebuttal 

AO group 35 1.37 1.00 3.46 1.69 53:51" $Sl2>SSI1** 

AM group 35 0.66 0.91 2.09 1.63 29.41*** $Sl2>SSI1** 

AN group 31 0.39 0.67 0.90 1.25 11.39* $Sl2>SSI1* 

OM group 38 0.82 0.69 2.16 1.82 24.69*** $S12>SSI1** 

ON group 35 0.49 0.66 1.91 1.58 41.46*** $Sl2>SSI1*** 

MN group 34 0.44 0.70 1.21 1.17 27.20*** $SI2>SSI1*** 
Qualifier 

AO group 35 0.14 0.36 0.26 0.61 2.80 

AM group 35 0.31 0.80 0.83 1.10 5.75* SSl2>SSI1* 

AN group 31 0.06 0.25 0.10 0.30 1.01 

OM group 38 0.24 0.59 0.92 1.40 12.60** $Sl2>Ssii™ 

ON group 35 0.06 0.24 0.17 0.57 2.80 

MN group 34 0.47 0.93 1.47 1.73 21:51" $Sl2>SSI1*** 


Note: “"p<.0001, “p<.001, “p<.01 


Table 4 presents that in general, the variables had statistically significant effects on the use of claims 
(A = .799, p < .001), warrants (4 = .837, p < .001), rebuttals (4 = .745, p < .001), and qualifiers (4 = .823, p< .001).A 
main effect analysis was therefore performed, and the results indicate that the type of combination affected the 
use of claims significantly only during the second (f,, = 9.98, p < .001) SSI debate. A Sidak post hoc test further 
reveals that the students in AO debates had better scores in terms of claims than the students who engaged in 
AN (MD = 1.09, p < .05), OM (MD = 1.93, p < .001), ON (MD = 1.80, p < .001), and MN (MD = 1.76, p < .001) debates. 
Moreover, it was also found that the students in the AM debates had better claim scores than the OM (MD = 1.15, 
p <.01), and ON (MD = 1.03, p < .05) debates. 


Table 4 


MANOVA Analysis of Four Arguments in the Two SSIs Debate Activities 


Claim 
Ssit 
ssl2 


Wilk’s A Multivariate F 
199 4.77" 
1.86 
9.98*** 


Univariate F 


Post Hoc Tests 


AO >AN group * 
AO >OM group *** 
AO > ON group *** 
AO >MN group *** 
AM >OM group ** 
AM > ON group * 


Warrant 
Sssit 


$sl2 
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837 3.73°* 
4.09** 


5.39*** 


AO >AN group * 
AO >OM group * 
AO > ON group ** 
AO >MN group * 


AO > ON group ** 
AO >MN group ** 
AM >OM group * 
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Rebuttal 745 6.37*** 
Sssi1 7.56*** AO >AM group ** 
AO >AN group *** 
AO >OM group * 
AO > ON group *** 
AO >MN group *** 
ssl2 11.14*** AO >AM group ** 
AO >AN group *** 
AO >OM group ** 
AO > ON group ** 
AO >MN group *** 
OM >AN group * 
AM >AN group * 
Qualifier 823 4.10*** 
Ssit 2.47* 
ssi2 8.40*** MN >AO group *** 
MN >AN group *** 
MN > ON group *** 
OM >AN group * 


Note: p<.0001, “p<.001, ‘p<.01 


As to warrant arguments, the type of combination had significant effects for both the first (Ff, = 4.09, p< .01) 
and second (f,,. = 5.39, p < .001) debate activities. A Sidak post hoc test shows that the students in the AO group 
had a significantly better warrant score than the students in the AN (MD = .83, p < .05), OM (MD =.71, p <.05), ON 
(MD = .86, p < .01), and MN (MD = .78, p < .05) debates for the first SSI debate. For the second SSI debate, the AO 
debate students still had better warrants than their peers in the ON (MD = 1.63, p < .01), and MN (MD = 1.58, p< 
.01) debates. For rebuttal arguments, the differences among the six types of debates are also statistically significant 
in both the first (f., = 7.56, p < .001) and second Mt = 11.14, p < .001) SSI debates. Sidak shows that the AO debates 
had a better score in terms of their use of rebuttals than the students in the other five debates for both the first 
and second SSI debates. As for the use of qualifiers, the type of combination had significant effects during both 
the first (f= 2.47, p < .05) and second i = 7.56, p < .001) SSI debates. Although the homogeneity of variance of 
the data was invalid for the first SSI i. 202) = 10.08, p < .001) and second SSI Ut Sis = 28.31, p < .001), the results of 
the Welch tests show the adjusted value was fine for both first (Welch = 2.38, p < .05) and second (Welch = 90.90, 
p <.001) SSI debates. 

The Follow-up Games-Howell post hoc test indicates that there was no statistically significant difference among 
the six groups in the first SSI debate. However, for the second SSI, the MN debate performed better than their peers 
in the AO (MD = 1.21, p < .001), AN (MD = 1.37, p < .001), and ON (MD = 1.30, p < .001) debates. Moreover, the OM 
debate had slightly better qualifiers than the AN debate students (MD = .82, p < .05). The analysis indicates that 
using qualifiers during the debate activities seemed to be a difficult task for most of the students, except for the 
students who engaged in the MN and OM debates. 


The Students’ Argumentation in the RJIs and Debates 


As to the student argumentation in the RJI. We focused on the groups and individual students whose perfor- 
mance scores were near the mean in order to explore the features of their argumentation. Follow-up response is 
an example of an M group student's (July) arguments in a cosmetic SSI interview. July shared her reasons for using 
and not using chemical cosmetics. 


There is different scientific evidence for supporting to use and not to use (chemical cosmetics). For example, people 
who agree with using cosmetics may say that it enhances our confidence. However, it actually can’t enhance our inner 
confidence. Here is an article | found on the internet in a research journal of 2002. The author, Rich, said that “upper- 
social women, especially adolescents and young adults, are the most dissatisfied with their bodies. They may use 
cosmetics to hide the things they do not like. However, their inner confidence is hardly to be increased.” 

As an M group student, she holds multiple points of view about the issue in both the first and second top- 


114 


https://doi.org/10.33225/jbse/24.23.104 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


THE EFFECTS OF STUDENTS’ STANDPOINTS IN ARGUMENTATION REGARDING SOCIO- ISSN 1648-3898 /Print/ 


SCIENTIFIC ISSUES ISSN 2538-7138 /0 firesy 


(pp. 104-118) 


ics of SSI. However, an improvement of her argumentation during the second topic of SSI was that she quoted 
statements from a research study to back her stance. Her arguments were sound and clear not only because the 
author and publishing year of the study were mentioned but also because the specific participants of the study 
were highlighted (upper-social adolescents, especially adolescents and young adults). In the analysis of the July's 
argument, we found that, first, she understood that to quote scientific information as backing would make her 
statement persuasive. Second, she knew how to quote scientific information as backing. Third, she also understood 
the authors’ statements were limited primarily to these “upper-social women”. Based on the analytical framework, 
the statement was regarded as a level 2 qualifier argument. 

The following qualitative example shows the students’ argumentation in the OA debate. 

1. Tina: cosmetics is one of the important factors to improve our economic growth. For example, in China’s 
and Turkey’s cosmetics market, cosmetic companies are continuing to expand rapidly. 
2.Abby: Youare right; | have related information about Turkey’s case; it is reported that the value of the beauty 
and personal care market in Turkish grew by 15.4%, about 3 billion in 2012. 
3. Lee: Maybe you are right, but | still believe that the most important aspect when making decisions about 
cosmetic issues would be the consideration of our environment. 

4. Susan: Yes, | agree with her (Lee). Here, | listed a number of environmentally damaging chemicals that are 
generally used in cosmetic products, such as synthetic fragrances, talc, and preservatives like butyl- 
paraben. The chemicals are recycled into our lakes, rivers and even into our food-chain systems. 

Tina, a student in the A group, stated that cosmetic products and companies play an important role for a 

country’s economic growth (No.1). A team member of Tina, called Abby, quoted a statement from a newspaper to 
support Tina’s argument (No.2). After that, an O group student (Lee) provided a disagreement (No.3). He respected 
the scientific information embedded in Abby's statement. However, he indicated that our environmental protection 
should be the first concern for making decisions regarding the cosmetic issue. Susan, a team member of Lee, agreed 
with Lee's point of view and pointed out a number of damaging chemical ingredients used in most of cosmetics 
(No.4). The episode shows that students gradually learned to support their assertions by using scientific information 
and evidence. One reason for explaining the improvement would be that they believe the use of scientific infor- 
mation would increase the persuasion of their statements. On the other hand, embedding scientific information 
in arguments could also be an effective strategy to make a response equal to these evidence-based arguments. 

As to the MN debate. They appeared to be productive in using qualifiers. The reason was that the M group 
students tried to point out and explain the pros and cons of SSI and expected their opponents (the N group stu- 
dents) to be more serious in discussing the cosmetic issue. 

5. Lee: Touse or not to use cosmetics is a personal affair. 

6. Helen: Yes, but if you don’t know how to use cosmetics, they will damage your skin. 

7. Sandy: there are several chemical ingredients that have been reported harmful to our skin, such as formal- 
dehyde, hydroquinone, and mercury. You should read the ingredient labels on cosmetic products. 

8. Vivian: Ifyou have to use them, please choose brands that adhere to independent certifications. Ifthe products 
were organic and only made with natural materials, we could use them. | share my favorite organic 
makeup brands to give your skin a healthy glow... 

At the beginning of the debate, the N team students had few comments about the cosmetic issue because 
they regarded it as a personal business (No. 5). Different from the N group students, the M group students (Helen, 
Sandy, and Vivian) used scientific information collected from newspapers and reports to persuade the N group 
students to be concerned about the issue seriously. They explained how to choose and use cosmetics, which is 
important to our skin health. Sandy indicated a number of chemical ingredients which are reported harmful to our 
skin (No. 7). Vivian provided a strategy to tell a cosmetic which was made with organic materials from these was not 
(No. 8). Sandy’s and Vivian's statements both are rational, and involved reasons of using and not using cosmetics, 
and thus, they were coded as a level 2 qualifier respectively in our analyses. They proposed level 2 qualifiers and 
tried to inform the N team students to select good cosmetic products before using them. 


Discussion 
The data from the debate reveals that the O group was more competitive in nature. In addition, most of the 


rebuttals produced by the O group were level 1 rebuttals, especially in the first SSI interview. This implies that 
these rebuttals tended not to be based on scientific evidence or theory. Mercer (2000) used the term ‘disputational 
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talk’ to explain students’ challenges and exchanges of rebuttals in argumentation. According to researchers, it has 
been found that the majority of middle school students possess the capability to offer rebuttals primarily based on 
personal opinions; however, rebuttals based solely on personal opinions do not contribute to students’ learning of 
argumentation, importantly, may cause conflicts and thus harm the classroom atmosphere (Albe, 2008; Lin, 2023; 
Zohar & Nemet, 2002; Zhu et al., 2020). This was an explanation of why level 2 arguments were rarely coded for 
the first interview of the O group. However, we considered that this tendency toward disputational talk should be 
seen as part of the process of students’ learning because our data also showed that the O group students made a 
significant improvement in terms of their scores for rebuttals and claims during the second SSI interview. 

It is worth noting that the M group demonstrated the highest overall argumentation performance among 
the four student groups, excelling in both the quality and quantity of the four types of arguments. These results 
can be explained by Kuhn's theory of the development of people’s epistemological understanding (Casas-Quiroga 
& Crujeiras-Pérez, 2020; Klopp & Stark, 2022; Kuhn et al., 2000). Among the four groups, the results showed that the 
N group students exhibited the least improvement and the least ability overall in their argumentation. Although 
the N group students did not perform well in their argumentation, their score in terms of the warrants they gener- 
ated was found to be as good as the scores of the other three groups. This implied to us that a teacher could guide 
the N group students to construct various arguments, i.e., claims, backings, rebuttals, and qualifiers, based on the 
knowledge and ways they construct warrants in order to create successful argumentation experiences for them. 

According to previous studies, the use of qualifier arguments poses a difficult task for most students because 
such usage emphasizes students’ reflections and evaluations of multiple standpoints on the given issue (Author, 
2022; Means & Voss, 1996; Schwarz et al., 2003). This kind of knowledge evaluation is, therefore, likely to take place 
in a debate with more exploratory talk rather than disputation or persuasion-related talk (Albe, 2008; Lin, 2023; 
Nussbaum, 2005; Osborne, 2005; Zhu et al., 2020). Based on this, the N group would combine more smoothly with 
the M group because their standpoint about the SSI was not as strong as those of the A and O groups, thus making 
the goal of the MN debate less about contradictory confrontations in which one student attempts to persuade 
others to accept his or her view and disagrees with any alternative objections, questions, or alternative claims. 
With less of a goal of persuading, the MN group students would have more chances to share their ideas, construct 
knowledge claims, and be curious about others’ ideas. This kind of learning atmosphere would provide opportu- 
nities for students to construct arguments from multiple points of view and, therefore, support the development 
of qualifier arguments. 


Conclusions and Implications 


An important result in the present study was that both the two factors: argumentation standpoint, and the 
combinations of two different argumentation standpoints for debate had significant effects on the frequency, as 
well as the quality of students’ using the four types of arguments, namely, claims, warrants, rebuttals, and quali- 
fiers in the SSI activities. Such effects increased in discussing the second SSI topic, indicating the more experience 
and knowledge regarding scientific argumentation students have, the greater the effect of their standpoint on 
their argumentation performance. The present study utilized two forms of argumentation in order to deepen our 
explorations of these issues in the SSI context: interviews and debates. The former form of activity emphasized the 
students’ within-group discussions and negotiations. We found that the A group students had the best performance 
in terms of producing claims and warrants among the four groups. Moreover, they produced fewer and weaker 
rebuttal arguments in both the RJI and debate activities. In contrast with the A group, most of the arguments 
generated by the O group were rebuttals and warrants. 

The statistical results indicate that all six types of student debates exhibited significant improvements in the 
utilization of claims, warrants, and rebuttals, whereas only half of the debate students, i.e., the students in the MN, 
MA, and MO debates, acquired significant improvements in term of using qualifiers. One obvious feature among 
those three combinations is the involvement of the M group students. As we discussed above, the M group stu- 
dents were the ones who exhibited the best ability in constructing qualifiers among the four groups because of 
their multiple perspectives on the issues. This could be a reason explaining their significant improvements. What 
makes us curious is why the MN group had the best improvement while the AM and OM groups had less significant 
improvement in terms of the use of qualifiers. 

These results suggest that teachers can investigate students’ standpoint on the learning topic of SSI and their 
prior knowledge about the standpoint before teaching. We also believe that students’ standpoints are not limited 
to the four types distinguished in this study, and such expectations can be further explored in future research. 
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Abstract. Students build up their STEM 
career interest based on their experiences. 
However, it remains unclear how students 

reflect on their STEM experiences in light 

of their understanding of STEM careers. 
This study aimed to explore how students 
relate their practices in STEM project-based 
learning (PBL) with their perceptions of 
scientists’ and engineers’ work. A randomly 
selected sample of students (n =142) 
participating in a STEM event participated 
in structured interviews regarding the 
resemblance between their months-long 

STEM PBL and scientists’ and engineers’ 

work. The data were coded using content 
analysis mostly by adopting a bottom-up 
approach followed by statistical analysis. 
Results showed that the majority of 
students claimed that their group had done 
things like scientists, while only about half 
of the students acknowledged doing things 
like engineers. The number and aspects 

of the students’ mentioned practices 

were generally limited, with engineer-like 
practices more divergent and reflecting 
their stereotype of engineers working as 
manual laborers. The results also suggest 
that students tend to neglect the minds- 

on but hands-off scientist- or engineer- 

like practices such as raising a question/ 
problem. The findings address the research 
gap regarding how students reflect on their 
STEM PBL experiences in light of career 
development. 
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Introduction 


STEM (the acronym for Science, Technology, Engineering, and Math- 
ematics) education is not only the aggregation of the four disciplines but 
also an integrated way of teaching and learning (National Research Council, 
2014; Office of the Chief Scientist, 2014). Generally, integrated STEM educa- 
tion has the potential to train students to solve real-world problems, enhance 
students’ understanding of how STEM disciplines benefit the world and their 
interest in STEM majors, and ultimately enable them to cope better with 
future challenges in their lives and careers. 

Students’ low aspiration level in STEM domains remains a concern in 
many regions worldwide (Marginson et al., 2013; OECD, 2019; Wang, 2017). 
Although many researchers have argued that STEM learning can foster stu- 
dents’interest (e.g., Kalogiannakis & Papadakis, 2020), studies have shown that 
engagement in STEM activities may not necessarily enhance STEM-related 
attitudes, including interest in STEM careers (Archer et al., 2014). How students 
interpret their STEM engagement and develop STEM career interests has 
been a focus of research in recent years (Lent et al., 2018). To date, researchers 
have found that students’ career development in STEM domains is a complex 
process that begins in childhood, and involves many internal factors including 
adequate perceptions of STEM careers (Garriott et al., 2016; van Tuijl & van 
der Molen, 2016) and STEM identity (Archer et al., 2013). 

Integrated STEM education emphasizes students’ active application of 
STEM subjects in authentic contexts (Papadakis et al., 2022). From a career 
development perspective, the authenticity of real-life problem solving in 
STEM activities has offered them career-related experience, and may theoreti- 
cally encourage students to think like a STEM professional, feel like a STEM 
professional, or consider whether or not they would pursue a STEM career. 
However, not much is known regarding how students interpret their STEM 
learning experiences with their understanding of STEM careers. The aim of 
this study was to explore whether and how students relate their practices 
in STEM learning with the work of scientists and engineers, and to examine 
patterns that emerged from students’ reflections. 
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Students’ Perceptions of Scientists and Engineers and STEM Identity 


Children begin to form a “map” of careers early in their lives (Gottfredson, 2002). However, many empirical 
studies have shown that students’ perceptions of scientific and engineering careers were often biased, limited, or 
were based on gender stereotypes (Fralick et al., 2009; Liu & Chiang, 2020). For example, Fralick et al. (2009) analyzed 
and compared 1,600 U.S. middle school students’ drawings of scientists/engineers and found that most drawings 
indicated males and scientists working indoors doing experiments, while there was limited understanding of 
engineers, with most existing perceptions involving performing outdoor manual labor. The findings of data from 
primary and middle school students in China were similar to the studies in Western contexts (Liu & Chiang, 2020). 
The gender stereotypes regarding students’ judgment of who are scientists (Miller et al., 2018) and who could be 
scientists (Banchefsky et al., 2016) were confirmed in later studies using different methods. In addition, empirical 
evidence suggests that stereotyped perceptions of engineers and scientists are resistant to change (Montfort et 
al., 2013), which may hinder their future career development. 

Students with more STEM career knowledge are more likely to pursue STEM careers (Blotnicky et al., 2018). 
Stereotyped perceptions of STEM careers may have a negative impact on adolescents’ STEM identity (Steinke, 
2017). A survey study involving over 3,500 students in Hong Kong SAR indicated that perceptions of engineers 
and career prestige mediated the positive effect of experiences in engineering learning on students’ engineering 
aspirations (Chan et al., 2019). Researchers have argued that students’ STEM career choices are influenced by the 
congruency and consistency among their perceptions of STEM careers, including gendered beliefs, career-related 
values, and identity (Wegemer & Eccles, 2019). 

The gender-related imbalance of achievement in STEM careers noted by many researchers (Ampartzaki et 
al., 2022) may be related to students’ perceptions of careers in this field. Students of different genders may have 
differences regarding perceptions of scientists’ and engineers’ work, which may further hinder their identity de- 
velopment. A quantitative survey study showed that compared to high school boys, girls’ knowledge of engineers 
and their working environments was poorer (Salas-Morera, 2021). Some researchers have collected students’ draw- 
ings of STEM professionals, and their results suggested that boys and girls may have different perceptions of what 
science and engineering are (Silver & Rushton, 2008). However, there are some contradictory results suggesting 
insignificant gender differences between boys’ and girls’ perceptions (Lampley et al., 2022; Liu & Chiang, 2020). 


Relating the Work of Scientists/Engineers by Engaging Students in Practices in STEM Learning 


There are both consistencies and inconsistencies between disciplinary science or mathematics learning in 
traditional classrooms and the processes of real problem solving that individuals are engaged in in their lives and 
at work. Some empirical studies have attempted to analyze the resemblance between science learning in school 
environments and the work of scientists. For example, Chinn and Malhotra (2002) found that commonly used 
inquiry tasks in school differ from scientists’ work on cognitive tasks, including whether students initiate their 
research questions, are provided with variables, and have clear directions for procedures. Zhai et al. (2014) found 
that students generally viewed themselves as “acting like a scientist” when they were doing experiments, while 
some other students thought they were not similar to scientists because they believed that scientists work alone 
doing dangerous tasks, and do not need to be well-behaved as students are in science classes. To summarize, 
the resemblance between classroom science and “real” science mainly lies in “doing” science, namely engaging in 
scientific practices. 

Another related concept is scientific and engineering practices, such as constructing explanations and design- 
ing solutions, which have been incorporated into national science curriculum standards in both the United States 
and China as prominent dimensions of learning (NRC, 2013; Ministry of Education, P. R. China, 2017), suggesting a 
shift from inquiry-based to practice-based learning and growing attention on engineering practices. According to 
the Next Generation Science Standards, these identified practices originated from “major practices that scientists 
employ” and “a key set of engineering practices that engineers use as they design and build systems” (NRC, 2014). 
It is expected that through engaging in these practices, students can relate disciplinary knowledge with its actual 
applications, understand how STEM disciplines are applied in real-world endeavors, and become aware of the 
careers in STEM domains (NRC, 2014). 

A concept broader than scientific and engineering practices is STEM practices. Develaki (2020) argued that 
scientific and engineering practices such as modeling and argumentation that are applied across STEM domains 
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can be viewed as “STEM practices.” These practices are embedded in integrated STEM learning in nature, accord- 
ing to Kelley and Knowles’s (2016) definition of integrated STEM education as “the approach to teaching the STEM 
content of two or more STEM domains, bound by STEM practices within an authentic context.’ 

To summarize, the resemblance between STEM learning and the work of scientists or engineers is manifest 
in practices that students engage in. The importance of engaging students in scientific and engineering practices 
has been widely recognized by the educational community (Jaber & Hammer, 2016). By engaging in scientific and 
engineering practices, students could take up “epistemic values and aims’ in science, comprehending what scientists 
and engineers seek to accomplish in their work (Chinn et al., 2011; Jaber & Hammer, 2016). Jaber and Hammer's 
(2016) analysis indicated that there is a consistency of emotions experienced within science practices between 
scientists doing science and students learning science. In addition, students with more engineering experience 
are less likely to hold stereotyped perceptions of engineers working as laborers (Chan et al., 2019), indicating that 
engagement in engineering practices may repel students’ naive perceptions of engineers. 


The Resemblance between STEM PBL and the Work of Scientists/Engineers 


Project-based learning (PBL) is one of the effective approaches that are widely applied (Barron, 2008; Krajcik, 
2014; Larmer, 2015; Thomas, 2000) to engage students in scientific and engineering processes. Compared to tra- 
ditional school science learning, PBL promotes goal-setting and ongoing constructive inquiries that are reflective 
and collaborative (Kokotsaki et al., 2016). Empirical evidence shows that STEM PBL can enhance students’ learning 
motivation (Bhakti, 2020), and attitudes towards STEM (Tseng, 2013), and promote their scientific creativity (Siew, 
2018). A great many informal STEM learning implementations take the form of PBL, which specifies no prescribed 
steps and is notimplemented in a rigid linear way. For example, in science fairs or competitions, students may often 
experience PBL; they form groups, initiate, and conduct investigations, or design tasks that resemble scientists’ or 
engineers’ work. 

STEM PBL provides opportunities for students to participate in a process that epistemologically resembles 
scientists’ or engineers’ work in knowledge construction and problem solving. Researchers have summarized a 
STEM framework consisting of the practices mentioned in STEM-related educational standards or significant docu- 
ments, and have used it to analyze students’ PBL reports presented at a science fair. The results showed that most 
of the analyzed STEM PBL groups performed practices in the STEM framework, such as defining the problem to 
be solved, creating and testing a solution, conducting an investigation, collecting data, and using technology (So 
et al., 2018). It can be reasonably hypothesized that, through engaging in STEM PBL, students may gain a deeper 
understanding of the work of STEM professionals’ use of science, technology, engineering, and mathematics in 
real-world problem solving. 

The resemblance between STEM PBL and the work of STEM professionals can be used in designing interven- 
tions with explicit instruction, making students realize that what they do in the classroom is similar to what STEM 
professionals do at work (Chen et al., 2022). For example, some interventions engage students in problem-solving 
activities that mimic those of STEM professionals, in which teachers explicitly inform students that they are doing 
tasks that certain STEM professionals, such as civil engineers or environmental scientists, deal with in their work 
(Beier et al., 2019; Chen et al., 2022; English et al., 2017; Kopcha et al., 2017). Empirical evidence suggests that these 
interventions explicitly relating STEM learning to STEM professionals’ work could enhance students’ understanding 
of STEM careers (Chen et al., 2022) and their STEM career aspirations (Beier et al., 2019). 

To sum up, the following arguments are summarized based on the literature review. First, students have lim- 
ited and biased perceptions of STEM professionals, which may inhibit their STEM identity development. Second, 
integrated STEM education has the potential to help students comprehend scientists’ or engineers’ work through 
practices. Last but not least, STEM PBL resembles scientists’and engineers’ work in practice, which may help students 
acknowledge the resemblance and develop their STEM identity through participating in STEM PBL. 

These arguments progressively reveal a significant research gap in how much resemblance between practices 
in STEM PBL and the work of scientists and engineers students can see after their STEM learning, especially when the 
learning involves little explicit career-related instruction. Addressing this research gap may enable STEM learning, 
including STEM PBL, to better provide opportunities for students to engage in processes similar to STEM profes- 
sionals’ work, which may increase their understanding of STEM careers, their STEM identity, and their aspirations. 

It is worth noting that the word “practice” used in this study was not confined to scientific and engineering 
practices in curriculum standards (for example, in NRC, 2013 and Ministry of Education, P. R. China, 2017) or STEM 
practices. Rather, we defined practices to be broader, to include the minds-on or hands-on activities or processes 
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that happen in STEM learning. Students perceived scientist- or engineer-like practices as being “larger” than scientific 
or engineering practices, such as the scientific inquiry or engineering design process; or it may be “smaller,” such 
as searching on the internet, or calculating means. Therefore, the broader definition is more helpful for analyzing 
students’ perceived practice in their STEM PBL experiences, which is not necessarily consistent with the scholarly 
definition in curriculum standards. 


Research Aim and Research Questions 


Based on the research gap identified above, the study aimed to explore whether and how students related 
their practices in STEM PBL to their understanding of scientists’ and engineers’ work. Specifically, two research 
questions were proposed as follows: 

RQ1:To what extent can upper-primary students relate the perceived practices in their STEM PBL to their 
understandings of scientists’ and engineers’ work? Are there gender differences? 

RQ2: In students’ reflections on their STEM PBL, what practices do students think they engage in that 
resemble scientists’ or engineers’ work? What are the patterns in these mentioned scientist- or engineer- 
like practices? 

Through the lens of perception of scientists’ and engineers’ work, this study aimed to understand how stu- 
dents epistemically comprehend the connection between their STEM PBL practices and STEM careers. The analysis 
may help educators make sense of students’ perceived practices in STEM PBL and how they build up their identity, 
self-efficacy, and career aspirations in STEM domains. 


Research Methodology 


As the research questions were exploratory in nature, it was more appropriate to gather qualitative data that 
allow for students’ flexibility in responding. However, since the research questions focused on general student 
epistemic understandings, adding quantitative analysis in addition to qualitative analysis was necessary to enhance 
generalization of the findings. Therefore, this study adopted a mixed-method approach, incorporating qualitative 
data as well as qualitative and statistical analysis, to address the research questions. In addition, interviewing 
techniques were used based on the consideration that the participants’ written expression may have been limited 
due to their grade level, as some were as young as fourth grade. Structured interviews that can reduce interviewer 
variability (Bryman, 2016) were performed to obtain comparable and quantifiable data, which can be more easily 
administered, aggregated, and analyzed, especially on relatively large samples for more generalizable findings. 
This study used both qualitative and quantitative methods to analyze the interview data. 


Sample 


The study collected data from the participants of a typical annual large-scale informal STEM event held in Hong 
Kong. Over 100 groups of primary school students participated in the event after they completed months-long 
extracurricular STEM PBL. Each participating group was made up of four to six students from fourth to sixth grade 
in one primary school. Student groups were responsible for initiating, conducting, and presenting the student-led 
STEM project under their teachers’ guidance (usually one to two teachers from the students’ school). The themes 
in students’ STEM PBL are widely diverse, ranging from developing and testing a solution or product to proposing 
and investigating a problem. The final project outcomes, including products and posters that describe their PBL 
processes, were presented by the student groups on the exhibition day, during which their projects were evaluated 
and awarded by the event organizers. 

A randomly selected sample of 142 fourth- to sixth-grade participants, coming from 72 project groups in 
the event from 68 primary schools, participated in the structured interviews. All participants were from schools 
in the Greater Bay Area in China, with most of the participating students (134) from Hong Kong and the remain- 
ing students from schools located in Guangdong Province. The grade and gender distribution of the students are 
shown in Table 1. 
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Information Regarding the Interviewed Students 


Grade Number of students Male Female 
4t grade 27 17 10 

5" grade 51 35 16 
6" grade 62 43 19 
Invalid 2 1 1 
Total 142 96 46 


Interview Design and Procedures 


Individual face-to-face structured interviews were conducted with each participant at the event by one of 
the four trained research assistants. The interview questions targeted the whole group (“your group members”) 
rather than only the interviewed student because, firstly, students collaborated and followed labor division in the 
whole group, and every group member was responsible for completing a particular part of the project; therefore, 
targeting each individual’s practices would be limited. Secondly, the students worked together to present their PBL 
on the event day for the awards. Hence, all participants were assumed to be very familiar with the whole group's 
PBL practices. 

The major interview questions are listed as follows: 

1. Have your group members done anything similar to scientists’ work in the project activities? 
2. If yes, can you describe the practices? 
3. Have your group members done anything similar to engineers’ work in the project activities? 
4. If yes, can you describe the practices? 

If the student gave an affirmative answer in response to the first interview question, the interviewer followed 
up by asking the second interview question. If the student gave a negative answer, he/she may give a reason for it 
as well. The same goes for interview questions three and four. All interviews were conducted by trained research 
assistants in the student’s native language. 


Data Analysis 


To address RQ1, which is quantitative in nature, a quantitative approach was applied to analyze the data gath- 
ered from interview questions 1 and 3. In addition to descriptive statistics, a paired-sample t test was performed 
on the data to analyze whether there was a difference between the likelihood of students acknowledging having 
scientist-like practices and engineer-like practices. Fisher's test, which is a nonparametric test for analyzing the 
correlation between two categorical variables, was performed to analyze the gender differences regarding whether 
students related their practices in STEM PBL to their understandings of scientists’'and engineers’ work. The analysis 
was conducted using the GraphPad Prism 9.0 software. 

To address RQ2, content analysis (Stemler, 2001), which can be used to analyze patterns and frequencies, 
was performed on responses to interview questions 2 and 4. Data were coded using a bottom-up approach, with 
most codes initially “emerging” from the data. Firstly, a draft coding scheme was developed by the first author 
after thoroughly reviewing the interview data, which was reviewed and revised by the corresponding author. The 
first author and a trained coder completed coding individually under the guidance of the coding scheme, during 
which the codes in the coding scheme were revised, re-organized, renamed, or combined for better parsimony 
and comprehensiveness, with reference to the STEM practices framework developed by So et al. (2018), which is a 
framework developed for and validated on upper-primary students participating in STEM PBL. The coding of the 
two coders was then compared, in which the differences were discussed until consensus was achieved on all the 
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coding among the coders and authors. To enhance the trustworthiness of the analysis, the draft coding scheme 
and codes were reviewed by the corresponding author to ensure that the codes were mutually exclusive and the 
coding was consistent with the data without over-interpretation. During the above-mentioned iterative process, 
the coding scheme was continuously revised, reorganized, and expanded until the coding was finalized. 


Research Results 


Percentages of Students Acknowledging their Practices as Being Like Scientists’ or 
Engineers’ Work and Gender Differences 


Most of the students in this study could relate their practices in STEM PBL to the work of scientists, while only 
about 60% acknowledged that their practices resembled the work of engineers. Of the students, 84.40% (119 of 
141, with one invalid response removed) responded that their group had done things similar to what scientists 
do, while only 15.60% believed that their group had done nothing like scientists (see Table 2). Moreover, 39.44% 
(56 of 142) claimed that there was nothing their group had done that resembled engineers’ work. Paired-sample t 
tests showed that the mean difference in the sample fell between the range of 0.140 and 0.342. There was evidence 
that the mean difference between whether students thought their group had done work resembling scientists 
and engineers was significant with t(140) = 4.707, p < .001. 

From a gender perspective (Table 3), male and female students’ responses acknowledging scientist- or engineer- 
like practices exhibited no significant differences. Specifically, 84.21% of boys and 84.78% of girls thought that they 
had done practices similar to scientists. Fisher's test indicated that there was no significant gender difference (p > 
.99). Regarding students’ understanding of engineers, 60.42% of boys and 60.87% of girls acknowledged that they 
had done things like engineers, while 39.58% of boys and 39.13% of girls believed not. Still, no significant gender 
difference (p > .99) was found using Fisher's test. 


Table 2 
Number of Students’ Acknowledging Doing Work Like Scientists and Engineers by Gender and Fisher's Test Result 


Acknowledging Not acknowledging BERTON ledging Not acknowledging 
Gender practices like ‘ : Pale p practices like ; : Ff p 
a practices like scientists : practices like engineers 
scientists engineers 
Male 80 15 58 38 
>0.99 >0.99 
Female 39 7 28 18 
Valid total 119 22 86 56 


Students’ Perceived Practices Resembling Scientists’ or Engineers’ Work and the Occurrences 


The average number of practices reported by students resembling scientists or engineers were 1.61 and 1.43 
respectively. The perceived practices mentioned by students with occurrences greater than 6% are shown in Figure 1. 

The practices most mentioned by students as being those of scientists seemed convergent, revolving around 
experiments and making. Among all the 142 interviewed students, 35.21% mentioned experimenting, such as do- 
ing specific experiments, going to the laboratory for experiments, conducting scientific experiments, and so forth, 
while 11.27% mentioned measuring (e.g., measuring pH value, electrical flow, temperature, and so forth). Making 
(e.g., making solar energy materials, making a gate, making colorimetric cards, and so forth) was also mentioned 
by 11.27% of the students. In addition, 9.15% of the students mentioned researching (e.g., researching how beans 
grow or how to reduce pressure on landfills). Repeating (e.g., keep testing and keep failing; repeating experiments) 
was mentioned by 6.34% of the students. 
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Figure 1 


Percentage of different practices resembling scientists’ work (Figure 1-a) and engineers’ work (Figure 1-b) mentioned by stu- 
dents (valid n = 142) 


(a) Cb) 
Experimenting 35.21 Making 13.38 
Measuring 11.27 Stereotypical actions 12.68 
Making 11.27 Designing 11.27 
Researching 9.15 Assembling 9.15 
Repeating 6.34 Measuring 8.45 
i i] 1 i} 1 i) i] i] 
0% 10% 20% 30% 40% 0% 10% 20% 30% 40% 
Percentage of students Percentage of students 


The most mentioned engineer-like practices by the students seemed more divergent than the scientist-like 
practices, generally with lower occurrences (Figure 1-b). The most mentioned engineer-like practice was making 
(13.38%). Students mentioned making materials, roofs, homemade tinfoil cups, and so forth in their PBL. Moreover, 
12.68% of the 86 students mentioned stereotypical actions. Here, stereotypical actions refer to skillful, manual, or 
operational practices that are not directly related to problem solving in STEM domains, such as drilling, cutting, 
stirring, or adding, while 11.27% of the students mentioned designing, such as designing the stand structure or 
designing how to make the whole system work. There were 9.15% of students who mentioned assembling (e.g., 
assembling a car, assembling a water pipe), and 8.45% who mentioned measuring (e.g., measuring the east-west 
and distance, measuring the size of the phone). 

Itis worth noting that, as for practices that are more abstract, more minds-on than hands-on, the occurrences 
are relatively low in the data. The occurrences of the students mentioning practices such as reasoning, testing hy- 
potheses, making a conclusion, designing an experiment, revising experimentation, applying technology, testing 
a design, reasoning, or analyzing the data were all lower than five (3% of the sample). What's more, there were 
no students who mentioned hands-off cognitive processes such as raising a question, argumentation, making 
hypotheses, or answering the question, either as scientist-like or engineer-like practices. 


Emergent Themes in Students’ Reflections on Scientist- or Engineer-Like Practices 


Some emergent themes were identified from the coding of students’ reflections on their STEM PBL practices 
resembling scientists’and engineers’ work. The identified themes reflected several aspects of students’ perceptions 
of scientists or engineers. 

In students’ responses that compared what they did in STEM PBL and scientists’ work, some students mentioned 
that like scientists, their PBL was also helpful, altruistic, and socially relevant. For example, the students said “[what 
we have done is] similar to scientists doing the same research in environmental protection” (B39-4, 6""-grade boy) and 
“Scientists have to create things that are convenient for the citizens” (B39-3, 6"-grade boy). These quotes suggest that 
some students saw the similarity between their goals in STEM PBL with those of scientists’ work. 

Many students seemed to reflect on their PBL with limited and superficial perceptions of scientists’ work. 
Although many students admitted doing things like scientists, some of them also added that the experiments 
they did were different from those performed by scientists because the location or methods were different from 
those of scientists. For example, one sixth-grade girl responded “But (our) experimental methods are different from 
those of scientists who work in laboratories, as we do not (work in laboratories)” (B27, 6"*-grade girl). For students who 
believed they did nothing like the work of scientists, their explanations usually exhibited a limited or superficial 
understanding of the work of scientists. For example, some students explained that they “do not understand what 
scientists do” (B30-4, 6"-grade boy) or “scientists do work in chemistry, but we do not” (B30-5, 5"-grade boy). 

Many students mentioned stereotypical actions as engineer-like practices, which may exhibit their superficial 


wu 


perceptions of engineers’ work. These practices, such as “connecting wires,’"“moving things around,’“cutting and 
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fusing,’“drilling,’ or“burning” describe very specific and skillful labor. Moreover, many students who believed their 


125 


https://doi.org/10.33225/jbse/24.23.119 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


STUDENTS’ REFLECTIONS ON THEIR SCIENTIST- OR ENGINEER-LIKE PRACTICES IN STEM ISSN 1648-3898 | /printy 
PROJECT-BASED LEARNING 
onsale ISSN 2538-7138 /onine) 


work was similar to that of engineers often had no clear concept of what engineers do, or confused it with the work 
of scientists or with people working in construction. Here are four quotes from four students: “! cannot imagine 
an example, but | am doing a similar job to engineering” (B54, 4‘*-grade boy), “ think it (engineers’ work) is similar to 
the work of scientists” (B55, 6th-grade girl),“(We) built a frame, which is like an engineer who starts a building and (we) 
went to buy materials to make it after designing it” (B33-1,6" grade boy), and “Some engineering departments which 
are responsible for building and need light at night can use our research. We can use our research to generate electricity 
and light at night” (B33-6, 5"-grade girl). 

Likewise, some of the students who believed their work was different from engineering also showed these 
stereotyped, biased, or limited perceptions of engineers: “Engineering is larger and belongs to buildings so has noth- 
ing to do with us” (B55-2, 6"-grade boy), “The work done by engineers is about science” (A21,5" or 6"-grade boy) or 
“| do not understand the definition” (B80-2, 5"°-grade boy). 

On the other hand, some students mentioned design-related practices, claiming that their group had engaged 
in designing products or designing solutions like engineers. For example, a few students mentioned “to design the 
generator and figure out how to build it” (B30-2, 5**-grade boys) or “to design how to make the whole system work” 
(B70-1, 5"°-grade boy). These reflections showed that a small number of students had a deeper understanding of 
engineering and engineers’ work and were able to reflect on their engineering practices in STEM PBL. However, 
some other students who acknowledged designing as engineer-like practices limited the design practices to 
building, designing, or model designing. 


Discussion 


The results of this study suggested that, even after long-term PBL participation, many students reflected that 
they had done limited/no practice resembling that of scientists or engineers in their STEM PBL. Unlike some previ- 
ous studies (e.g., Luo & So, 2023; Salas-Morera, 2021; Silver & Rushton, 2008), the results indicated no significant 
gender differences. Furthermore, identified themes in students’ mentioned scientist- or engineer-like practices 
with previous findings regarding students’ perceptions of scientists and engineers have been compared in the 
Discussion section. 


Students’ Limited Reflection of Scientist- or Engineer-Like Practices in STEM PBL 


The percentage of students believing their group had done nothing like engineers or scientists and the low 
average number of practices mentioned by the students suggested that their reflections of scientist- or engineer- 
like practices in PBL were confined. The low occurrence of practices that are not directly related to hands-on prac- 
tices, such as raising a question or solving problems, implied that very few students could recognize the epistemic 
resemblance of these minds-on practices between STEM PBL and scientists’ or engineers’ work. 

This may be due to students’ limited knowledge of the epistemic nature of science and engineering or, in other 
words, scientists’ and engineers’ work. A previous quantitative study on upper-elementary Hong Kong students 
who participated in STEM PBL showed that most students acknowledged that they had done these minds-on STEM 
practices such as making hypotheses, drawing conclusions, and designing models to solve a problem (So et al., 
2018). Based on these previous findings and evidence from this study, it is plausible to hypothesize that students 
are aware of the practices they engage in in STEM PBL, but they often cannot relate what they do to scientists’ or 
engineers’ work because of their limited knowledge of such work. Therefore, career-related information could be 
introduced in ways such as inviting STEM professionals to class (Hopwood, 2012), presenting scientists’ biographies 
(Lessard, 2011), or exposing students to authentic learning environments (Singer et al., 2020). 


Naive Perceptions of Scientists and Engineers in Students’ Reflections on STEM PBL 


The results of the study echo previous research regarding students’ naive, superficial perceptions of engineers 
and scientists (for example, Fralick et al., 2009, Lachapelle et al., 2012; Liu & Chiang, 2020). Specifically, some students’ 
reflections regarding their practices in PBL viewing figurative manual activities as engineer-like practices echo pre- 
vious findings that some students have similar misconceptions about engineers, and think of them as blue-collar 
skilled workers (Ergun & Balcin, 2019; Jordan & Snyder, 2013; Karatas et al., 2008), implying that students’ beliefs 
regarding what they had done like engineers were confined to their limited or biased perceptions of engineers’ 
work. A possible explanation for these patterned naive perceptions of engineers is that students do not have a deep 
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understanding of engineering, but rather have a superficial impression of engineers’ practices (Lachapelle et al., 
2012). Students tend to understand engineers’ work through the “superficial” hands-on aspects but not the cogni- 
tive aspects of their work, which may lead to confusing engineers with skilled workers (Cunningham et al., 2005). 

The data analysis suggests that students’ acknowledged scientist-like practices in their STEM PBL are more 
realistic and convergent, while some engineer-like practices reflected their stereotypical perceptions of engineers’ 
work. This finding supports some previous studies (for example, Fralick et al., 2009; Luo & So, 2023) in that students 
perceptions of scientists are more accurate than their perceptions of engineers. However, some students believed 
they had done nothing like scientists because of their non-laboratory working location or specific methods. This 
superficial understanding of scientists’ work echoes the finding of Zhai et al. (2014) that upper-primary students 
view their practices in science learning and scientists’ work differently. 


O 


Conclusions, Limitations, and Implications 


To sum up, the results of this study indicate that even when upper-primary students were deeply involved in 
STEM PBL for months, a small number of students believed that they had done nothing like scientists, while nearly 
half of them thought they had done nothing like engineers. These perceptions showed no significant gender dif- 
ferences. The upper-primary students’ descriptions of their scientist-like practices in PBL mostly revolved around 
experiments and research. The engineer-like practices mentioned by the students were more divergent, with 
some reflecting students’ stereotypical views of engineers as doing skillful labor. Moreover, the students seemed 
to neglect practices that are minds-on and hands-off when reflecting on their scientist- or engineer-like practices 
in PBL. The above conclusions are illustrated using a figure in the appendix. 

The study does have several limitations that should be noted. First, the interviewed students were participants 
in a STEM event, which possibly indicates they have a higher interest in STEM domains than average students. 
What's more, although the study design involved structured interviews and a relatively large sample to improve 
generalizability, the findings should be interpreted with caution when extrapolating to students of different age 
ranges or cultural contexts. Future studies could investigate the applicability and generalizability of the findings 
with samples of larger populations and student diversity. 

The findings of this study have both theoretical and practical implications. The study addressed the research 
gap by examining students’ reflections on scientist- or engineer-like practices in STEM PBL, which are rarely explored, 
thus filling a theoretical void. The present findings extended prior research by suggesting that students’ naive 
perceptions of scientists and engineers’ work may inhibit students’ identity formation by limiting their reflections 
on their PBL experiences. 

As regards practical implications, the findings emphasized the value of explicit instruction regarding career- 
related reflections in STEM learning, indicating that, without instruction, students may not “naturally” realize 
the resemblance between their practices and STEM endeavors in the real world, even after long-term STEM PBL 
engagement. Future interventions that integrate career information with STEM activities are suggested, as they 
may repel stereotypical perceptions of STEM professionals and help students relate their STEM engagement with 
real-world STEM endeavors and careers, through which students may better develop STEM-related identity and 
career interest derived from their own experiences. 
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Abstract. /n light of the increasing 
frequency of misleading information in 
social media regarding environmental 
issues, this study aimed to identify 
misleading information spread through 
TikTok videos and to discuss why such 
content is considered misleading, drawing 
on relevant literature. Hashtags with large 
numbers of views, such as #climatechange, 
#sustainability, #pollution, #biodiversity, 
#environmentalprotection, 
#environmentalissues, #energysource, 
and #environmentalproblems, were used 
for data collection through web scrapper 
called Apify (https://apify.com/). A total of 
29 misleading videos were found. Content 
analysis was applied to identify and classify 
the topics and misleading claims. The 
topics of misleading videos, according to 
the most frequent mentions, were energy 
sources, followed by climate change, 
pollution, biodiversity, and environmental 
degradation. Among the misleading 
claims, videos related to pyramids as non- 
pollutant power plants and conspiracy 
related to pollution exhibited the highest 
frequency. The results show various 
misleading claims in videos related to 
environmental topics. Also, emphasized 
the importance of science education 

in addressing misleading information. 

In addition, the importance of an 
interdisciplinary approach for addressing 
environmental issues was reinforced. 
Keywords: TikTok videos, misleading 
information, environmental issues, content 


analysis, science education 
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Introduction 


In recent years, with the advancement of information technologies and 
social media, easy access to information has a problem, that is, the propaga- 
tion of misleading information. As pointed out by Petratos (2021), misleading 
information can occur in different forms such as misinformation, disinforma- 
tion, or fake news, which is easily spread through social media. Flintham et 
al. (2018) have pointed out that the advent of social media changed the con- 
sumption and exposure to various news. Aldwairi and Alwahedi (2018) also 
emphasized that fake news is an old concept; it has existed since publishers 
first used misleading information to further their interests even before the 
Internet. However, the advent of the web further increased and redefined it. 

In general, different types of misleading information have been influ- 
encing diverse areas of society, such as health (Pulido et al., 2020), politics 
(Farkas & Schou, 2019), and the sciences (Scheufele & Krause, 2019). In terms 
of environmental topics, misleading information currently targets climate 
change (Lutzke et al., 2019), global warming, and the influence of green- 
house gas emissions (Lopez & Share, 2020). These topics emerge especially as 
doubts about human influence on problems such as climate change, increase 
in greenhouse gas emissions and global warming. This spread of misinfor- 
mation can be disastrous during environmental crises, which emphasize 
the importance of drawing attention to misleading information related to 
environmental topics spread on social media. 

In the context of misleading information, education can be viewed as 
an essential tool for enabling society to address this current issue. Previous 
scholars have pointed out the importance of education for coping with fake 
news (Black & Fullerton, 2020; Keselman et al., 2021; Santin, 2021). Specifically, 
in the case of the environment, environmental education plays an important 
role and can be considered a strategy for changing attitudes toward the future 
(Carvalho, 2001; Latif et al., 2013; Novo, 2009; Varela-Candamio et al., 2018; 
Zs6ka et al., 2012). Following this notion, Breiting and Mogensen (1999) high- 
lighted that the major focus of environmental education is teaching about 
“the environmental issues man faces through his use of natural resources and 
the possibilities of overcoming and preventing them in the future” (p. 349) 
instead of the environment. 
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In the context of science, Scheufele and Krause (2019) shed light on a few probable causes of misleading infor- 
mation on scientific topics, such as the “lack of understanding of science,’ which includes “knowledge about scientific 
facts” and “epistemic knowledge about science.’ Other examples include“holding beliefs inconsistent with the best 
available science” such as “inaccurate views of scientific consensus and the willful rejection of scientific consensus” 
and “conspiratorial beliefs.” In this sense, science is a fundamental tool for confronting misleading information. 

The increase in the spread of misleading information gained scholarly attention toward the detection of 
fake news mainly with the use of algorithms (Aldwairi & Alwahedi, 2018; Alonso-Ldpez et al., 2021; Flintham et al., 
2018; Shu et al., 2017). Additionally, many social media platforms have implemented policies against the spread 
of fake news. For example, Meta, the company who owns Facebook, Instagram and Whatsapp, has been removing 
content that violates its policies, which includes fake news and misleading content (Combating Misinformation | 
About Facebook, 2020). For climate change, the company also clarified that despite the topic being part of a low 
percentage of fake news on the platform, any fake news is taken very seriously (Meta Sustainability, 2022). Toward 
this end, it partnered with a network of more than 90 independent fact-checking leading entities to evaluate and 
rate climate-related content in more than 60 languages. Content found as false receives a warning label and obtains 
reduced visibility. In addition, pages, persons, groups, or ads that constantly share false information are penalized. 
However, in terms of controversial issues, the position of the platform is described as follows: 


For example, content may use true information or opinion to express uncertainty about the impacts of climate change, 
distrust in scientific expertise or skepticism about climate solutions. We don't believe it is our place as a company 
to penalize this type of speech or referee legitimate debate, which is why we take the approach of educating and 
informing people with authoritative information (Meta Sustainability, 2022, para. 8). 


Other platforms, such as Instagram, follow similar steps, because they are managed by the same company, 
Meta (How Instagram addresses false info, n.d.). Users of Instagram and Facebook also can report misinformation, 
and the company will verify if the claim is valid. The former Twitter current X, also uses a labeling information tool 
that alerts it when information can be misleading (How We Address Misinformation on X, n.d.). The labeled infor- 
mation, similar to the other platforms, is penalized with reduced visibility. 

For Tiktok, Keenan (2022), the head of trust and safety of the platform, shared that they are working to im- 
prove measures for reducing misinformation on the platform. For example, they use machine learning to detect 
misinformation, which is redirected to a moderation team that analyzes and removes it in the case of violation of 
their policies. The platform also relies on fact-checking partners that review content in more than 30 languages. 
In a recent update, Tiktok (2023) informed that it will also begin to enforce measures against the spread of misin- 
formation on climate change. 

Among these social media, Tiktok has been gaining considerable attention recently. In 2020 and 2022, it was 
the most downloaded app worldwide (Ditrendia, 2020; Most Downloaded Apps Worldwide 2022 | Statista, 2023). 
In addition, as pointed out by Alonso-Lépez et al. (2021) and TikTok: Distribution of Global Audiences 2023, by Age 
and Gender (2023), Tiktok’s userbase mainly consists of Generation Z. The platform was launched locally in China 
in 2016 and internationally later in 2017. In 2018, it merged with Musical.ly and became accessible in the United 
States (Zulli & Zulli, 2022). As explained by Hautea et al. (2021), Tiktok features videos lasting for 60 s, oftentimes 
even shorter, “the in-app interface is used to record and edit video content, annotate it with text and graphics, and 
post with captions and hashtags” (p. 3) based on likes, shares, and comments, and other interactions, which guide 
the recommendation algorithms of the platform. However, tracking misleading videos uploaded on the platform 
can be difficult due to its popularity, ease of features, and large number of videos uploaded per day. Moreover, 
compared with other social media, Tiktok can still be considered new; thus, examining its contents from the plat- 
form can lead to interesting results. 


Theoretical Background - Misleading Information 


According to Collins Dictionary (n.d.), misleading information is a type of information that can confuse or 
mislead. It can also be defined as a type of information that can be considered deceptive. In the literature, various 
fields of research focus on defining this type of information. For example, Wardle and Derakhshan (2017) high- 
lighted the definition of terms and strategies as policy-making to combat misleading information or information 
disorders. Lecheler and Egelhofer (2019) focused on the terms and impact of misleading information. Brisola and 
Doyle (2019) emphasized terms and proposed the critical information literacy approach for addressing this type of 
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information. Bailey and Hsieh-Yee (2019) examined the history and terms and proposed a framework for addressing 
false information through information literacy. 

In addition, various studies on misleading information can be found in the literature, and the number of studies 
on the topic is continually growing. For example, Ha et al. (2021) found that research on misleading information 
is increasing. Specifically, their review demonstrated a spike in the number of studies on misinformation and fake 
news after 2016. Moreover, the review highlighted that quantitative research was the most commonly used method 
and that diversity exists in the disciplines that conducted research on this topic at the time. 

The majority of research on misleading information is still related to computer science (Ha et al., 2021). In this 
field, many researchers proposed an algorithm detection for fake news on social media. (Figueira & Oliveira, 2010; 
Natarajan et al, 2022; Shah & Kobti, 2020; Sheikhi, 2021; Shrivastava et al., 2022). Guo et al. (2020) reviewed the dif- 
ferent methods used for the detection of false information on social media. Among other analyses, Zannettou et 
al. (2019) also considered the detection of false information in which the authors found that the most commonly 
used method was machine learning. When analyzing misleading information on social media platforms, Zanettou 
et al. (2019) and Kumar and Shah (2018) cited that the majority of researchers used Twitter followed by Facebook 
as the object of study. 

As mentioned by West and Bergstrom (2021), the majority of research on misleading information is related to 
the public consumption of misinformation, models of spreading misleading content, social network effects, and 
crowd-sourced mediation. Scholars also examined the strategies for addressing misinformation, one of which is the 
inoculation method, which has produced interesting results. As explained by Lewandowsky and Van Der Linden 
(2021), the inoculation method, which is analogized to inoculation for vaccines, consists of the idea of exposing 
people to weak examples of misleading information beforehand, such that they become increasingly resistant or 
immune to such misinformation. Many researchers have used this strategy (Compton et al., 2021; Lewandowsky & 
Van Der Linden, 2021; Maertens et al., 2021; Roozenbeek et al., 2022; Van der Linden, 2017). 

Scientific topics are also frequent targets of misinformation. For example, as illustrated by Wang et al. (2019), 
health topics, such as vaccines, cancer, cardiovascular diseases, diet, and nutritionism, are constantly the target of 
misinformation. Del Vicario et al. (2015) conducted a study in Italy on the environment, diet, health, and geopolitics, 
which are the most consumed conspiracy topics on Facebook. 

Climate change is another topic that has been examined in relation to misinformation. Treen et al. (2020) 
pointed out that misinformation on climate change is related to skepticism, denial, and contrarianism. The authors 
also highlight the importance of interdisciplinary strategies to deal with this problem. Other studies on misinfor- 
mation about climate change have also been conducted. For example, Samantray and Pin (2019) analyzed climate 
change conversations on Twitter, while Wright (2022) conducted a review on misinformation on climate change 
and extreme weather across social media platforms. Lastly, Chu et al. (2023) explored the semantic features of 
climate change on Weibo. 

As a result of scientific topics being frequently the target of misleading information, many researchers point 
out the importance of science in combating misinformation. Scheufele and Krause (2019) explained different rea- 
sons why citizens are misinformed such as the lack of understanding of science. In a different context, Hopf et al. 
(2019) associated the propagation of misinformation with the lack of trust in science. Various researchers, such as 
Sharon and Baram-Tsabari (2020), Howell and Brossard (2021), and Serpa et al. (2021) pointed out scientific literacy 
as a means of addressing misleading information. 

Moreover, education is a seemingly positive influence on addressing misinformation. Brisola and Doyle (2019) 
emphasized that critical information literacy and education are tools for solving this problem. Other researchers, 
such as Hopf et al. (2019), also emphasize that education is important in this process. The authors proposed that 
education, as part of life skills development and especially in the culture and methods of science, is an important 
component of long-term solutions for misleading information. 

In the specific case of science education, science literacy is one of the goals of the science curriculum in dif- 
ferent countries as mentioned by Osborne and Pimentel (2023). For the authors, the science curriculum is failing in 
addressing misinformation and promoting appropriate scientific literacy. They also propose basic competencies for 
evaluating scientific content, which are related to an enhanced understanding of science and its nature. In another 
paper, Osborne and Pimentel (2022) highlighted the role of education in the age of misinformation. They posited 
that education plays an important role in well-known strategies for handling misinformation such as inoculation, 
debunking, or lateral reading. The authors also named three concepts that are important for understanding and 
evaluating misleading claims. Science education can be helpful in the use of the three concepts, namely, “(i) the 
social practices that the scientific community uses to produce reliable knowledge (10); (ii) the criteria of scientific 
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expertise; and (iii) the basics of digital media literacy.’ (p. 247). The researchers emphasize the importance of 
prioritizing science literacy in education. They address misinformation by proposing fundamental strategies and 
underscore the significance of education in instilling critical concepts for assessing scientific content in an era of 
misleading claims. 


Research Problem 


A stream of research has examined climate change on social media; however, research that explores beyond 
this topic in environmental issues remains lacking. On Twitter, for example, Samantray and Pin (2019) analyzed 
conversations about climate change denial, and Fownes et al. (2018) conducted a review on conversations about 
climate change on social media. Furthermore, other researchers created datasets of opinions about climate change 
on Twitter (Effrosynidis et al., 2022).On Facebook, scholars focused on testing different approaches for evaluating the 
credibility of fake news (Lutzke et al., 2019), analyzing contents created by non-government organizations (NGOs) 
(Vu et al., 2020), and examining rhetoric on climate change denial on social media (Bloomfield & Tillery, 2019). On 
Instagram, studies explored how climate change communication is implemented on the platform (Kober, 2022) or 
analyzed the content of the hashtag #fridaysforfuture (Herrmann et al., 2022). On Tiktok, research was conducted on 
the hashtag #climatechange (Basch et al., 2022), and semantic network analysis was used to investigate hashtags 
such as #climatechange, #ecofriendly, #sustainability, #ecotok, #forclimate, and #environmentalactivism (Nguyen, 
2023). However, other topics of environmental issues (apart from climate change) remain under-explored, and the 
majority of research is still quantitative. Furthermore, an in-depth analysis beyond the number of likes and views in 
Tiktok is necessary. Expanding the understanding of misleading content on social media platforms becomes crucial 
in the international context, particularly given the importance of science education in mitigating misinformation. 
Osborne and Pimentel (2023) have highlighted challenges within current science curricula, emphasizing the need 
for innovative approaches. Studies exploring scientific content on social media platforms provide valuable insights. 
Beyond merely detecting misinformation, understanding the topics used, their arguments, and misleading claims 
can greatly benefit the global science education curriculum. However, as noted, the analysis of misleading infor- 
mation on social media still lacks in-depth research. Based on the literature review in this study, no research has 
explored the different types of misleading information on the aforementioned platforms. 


Research Aim and Research Questions 


This research aimed to analyze the nature of misleading content found on Tiktok videos and to discuss why 
such content is considered misleading through the relevant literature. Especially, there was an intention to identify 
misleading content in videos on Tiktok regarding environmental issues, analyzing it by topic, and identifying specific 
misleading claims in these videos. Thus, the study posed the following research questions. 

1. What are the main topics of misleading videos related to environmental issues in Tiktok? 
2. What are the misleading claims made in these videos? 
3. What does the relevant literature say about these topics and misleading claims? 


Research Methodology 
General Background 


This research is qualitative because it aims to analyze the nature of misleading content in TikTok videos on 
environmental issues and understand why it is considered misleading. This type of research aligns with qualita- 
tive research approaches that intend to examine the intentions and meanings underlying the data, exploring 
meanings and generating insights instead of focusing on statistical analysis. Data were collected using hashtags 
(#climatechange, #sustainability, #pollution, #biodiversity, #environmentalprotection, #environmentalissues, 
#energysource, and #environmentalproblems) through a web scraper called Apify (https://apify.com/) on June 
2, 2023. For the data analysis, qualitative content analysis by Kripendorff (2004) was used. This analytical method 
is frequently used to analyze textual, visual, or audio data to identify themes, patterns, and meanings within the 
content. After sampling materials, speeches, and text (if presented and if different from speech) were transcribed. 
These data transcriptions were used to build categories to identify the general topics of the videos and each mis- 
leading claim presented in them. 
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Sample 


The sample consisted of 3,840 videos posted on Tiktok from 2019 to 2023. All videos that were private, un- 
available at the time, or unrelated to environmental issues were excluded. The remaining videos were considered 
relevant. The total number of relevant videos was 2, 198, which were watched again to identify misleading informa- 
tion. To identify potential misleading information, the current study used a method similar to that used by Sharma 
et al. (2017) and based the analysis on the quality of information and citation of credible sources. All videos that 
provided doubtful information were separated for fact checking. A total of 40 videos presented information that 
was considered doubtful, which was examined to verify the contents on the basis of the relevant scientific literature. 
Out of 40, 29 were considered misleading. The videos and their specific data were grouped under three categories 
(i.e., relevant, misleading, and non-misleading) and systematized using Excel sheets. The average duration of the 
misleading videos was 2.1 min. Table 1 presents an overview of the number of views, likes, comments, and shares 
of the misleading videos. 


Table 1 
Overview of Tiktok Videos Related to Environmental Issues 


Videos re Total views Total likes Total comments Total shares 


Misleading 29 19,405,383 2,437,860 35,703 116,538 


Table 1 indicates a significant amount of engagement with misleading videos, which can be seen by the total 
high number of views, likes, comments and shares. Thus, analyzing their content is relevant. 


Data Collection and Analysis 


For data collection, Tiktok videos were assessed using hashtags. The method used by Basch et al. (2022) and 
Huber et al. (2022) was followed and included the hashtags #climatechange and #sustainability to identify videos 
related to environmental issues. An additional reason for including these hashtags was that they displayed the 
highest number of views (5.3 and 4.6 billion views, respectively) at the time of data collection compared with the 
other hashtags identified as relevant by the current authors. 

Furthermore, based on these two hashtags and number of views, additional hashtags were derived using 
snowball sampling (Noy, 2008) as follows: #pollution (1.1 billion), #biodiversity (263.7 million), #environmentalpro- 
tection (152 million), #environmentalissues (3.8 million), #energysource (2.5 million), and #environmentalproblems 
(1.7 million), which were also included in the analysis due their relevance and high number of views. 

Data from these hashtags were extracted through a web scraper called Apify (https://apify.com/) on June 
2, 2023. Apify enables data scraping from web pages and has been used by many researchers to examine videos 
from social media platforms (e.g., Dubrosa et al., 2023; Nallakaruppan et al., 2023; Wu et al., 2023). According to 
the definition of the Apify website, web scraping is a method for automatically extracting data from webpages 
and other online sources. It entails accessing online pages, parsing the HTML content, and retrieving particular 
data—like costs, descriptions or locations—using software tools or scripts (Explore Web Scraping Tools, Methods, 
and Frameworks, n.d.). 

Specifically, for this research, Apify was used to retrieve data from TikTok videos related to environmental issues 
and the mentioned hashtags. Through Apify, data such as the number of likes, views, shares, comments, and the URL 
of the videos were collected. The current study is interested in the content of misleading videos; thus, all contents 
of the 29 videos were transcribed into a Word document. Each video was identified using the link of the video 
and the code used for identification of the videos (e.g., V1 and V2). The qualitative content analysis with reference 
to Krippendorff (2004) was employed, this method is defined as “a research technique for making replicable and 
valid inferences from texts (or other meaningful matter) to the context of their use” (p. 18). This method appeared 
to be suitable for the aims and questions of this research, providing a flexible, exploratory, and in-depth approach 
to understanding misleading content in TikTok videos related to environmental issues. Given the exploratory na- 
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ture of this study, aligned with its aims and questions, qualitative content analysis is deemed appropriate. Several 
studies have utilized content analysis for exploring videos, indicating their validity and usefulness for this type of 
research (Fowler et al., 2021; Kousha et al., 2012; McCashin & Murphy, 2022; Waters & Jones, 2011). First, the videos 
were categorized into their main topics in relation to environmental issues. The categories were created on the 
basis of the topics for environmental issues as discussed in the book of Haper and Snowden (2017). Second, each 
misleading claim that appears in the videos was categorized. Table 2 systematized and presented these data. The 
researchers discussed all topics and misleading claims using scientific papers and books as references. 


Research Results 
Topics of Misleading Information 
The analysis focused on the content presented in TikTok videos related to environmental issues that contained 
any type of misleading information. Furthermore, the results were organized based on the main topic of the videos 
and the misleading claims made in each one. The summarized results regarding the main topics of the videos and 


each misleading claim are presented in Table 2 as follows. 


Table 2 
Topics on Environmental Issues and Misleading Claims Presented in Misleading Tiktok Videos 


Topic ae Misleading Claims ae 
Non-pollutant energy source: Free fuel generator of energy 1 (2.9) 
Dark energy as a source of clean energy 1 (2.9 
Non-pollutant energy source: Pyramid as a power plant and Nikola Tesla 8 (23.5) 
Energy sources 12 (41.4) | Renewable energy is too expensive 1 (2.9 
Renewable energy sources are not reliable 1 (2.9) 
More people are killed installing solar panels every year than are killed by nuclear power 1 (2.9 
CO2 has no influence on climate change 1 (2.9) 
Earth is cold most of the time 1 (2.9 
We can live on Earth when it is warm 1 (2.9 
The Earth is warmed up without human intervention 1 (2.9) 
Climate change 7 (24.1) The future of the climate is going to be decided in Asia and in Latin America by poor people 2 (5.9 
The source of climate change is a tire graveyard 1 (2.9) 
Investments won't help to reduce global temperatures 1 (2.9 
Earth goes through cycles and, therefore, it is warmed up without human intervention 1 (2.9) 
zeal 1 (3.4) People from Mars suffered from global warming 1 (2.9) 
Biodiversity 2 (6.9) Rating animal appearance: Beauty and ugliness of animals 2 (5.9) 
Three-eyed fish found in Argentina mutation due nuclear waste 1 (2.9) 
Pollution 7 (24.1) 
Conspiracy related to pollution: Disbelief in government, companies, science, or news 8 (23.5) 
Total 29 (100) Total 34 (100) 


Table 2 indicates a diverse range of topics related to misleading content on Tiktok and presents each mis- 
leading claim by topic. Moreover, the topics are related to the main topic of the video. Misleading claims have a 
higher number of topics because some videos presented more than one misleading claim. The topics with the 
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most misleading content were Energy source (n = 12, 41.4%) followed by Climate change (n = 7, 24.1%), Pollution 
(n = 7, 24.1%), Biodiversity (n = 2, 6.9 %) and Environmental degradation (n = 1, 3.4%). 

The results show that among the misleading claims the ones with higher mentions in the videos were Non- 
pollutant energy sources: Pyramid as a power plant and Nikola Tesla (n = 8, 23.5%) and Conspiracy related to pollution: 
Disbelief in government, companies, science, or news (n = 8, 23.5%). These claims were succeeded by others, such 
as The future of the climate is going to be decided in Asia and in Latin America by poor people (n = 2, 5.9%) and Rating 
animal appearance: Beauty and ugliness of animals (n = 2, 5.9%). Other misleading claims spread in fourteen differ- 
ent claims were mentioned only once (2.9%). 


Energy Source 


This topic is the most mentioned one. The videos were related to green energy sources, the idea of a fuel-free 
generator, dark matter as a source of energy, and natural energy from pyramids as well as Nikola Tesla. Out of the 
12 videos related to energy sources, eight were related to pyramids being a power generator that produces clean 
and non-pollutant energy. In addition, among the misleading claims about energy sources, one of the videos was 
related to the denial of climate change, the influence of CO, and global warming. Overall, the video was about 
renewable energies, but they mentioned a misleading claim related to the denial of climate change. The following 
text presents examples. 

The first is the idea of clean energy produced by pyramids and its relation to the work of Nikola Tesla. 


He discovered that electricity occurred naturally throughout the Earth’s atmosphere and ground. This energy was 
not the kind of energy that we have that pollutes. It was a completely passive energy. That had no byproducts. (...) 
Carmen, can you explain to me why do you think the Great Pyramids of Giza in Egypt are not tombs? It has nothing 
to do with that. And they keep repeating. They get a story, and they repeat it over and over, and they get the students 
to repeat it, to get their mark to get out of college. And none of it. They have no proof of any of it. They never found a 
mummy ina pyramid. Why do you think they were power generators? Christopher Dunn has shown how the pyramids 
could have been used as energy plants. He’s written several books that he’s a power plant. He’s an engineer. And so 
his research isn’t psychic and, you know, esoteric in any way. It’s mechanical. (...) (V20, audio transcription) 


First, it is important to note that the assertion suggesting that the Great Pyramid is not a tomb lacks support 
from relevant literature. Theories, such as the Great Pyramids being energy power plants, are considered alterna- 
tive theories and are not widely accepted by experts in the field. The claim that students are forced to accept and 
repeat that the pyramids are just tombs to pass exams is seemingly conspiratory. Schools and teachers bear a moral 
responsibility to teach students the most accepted knowledge in society. Lastly, the association between Nikola 
Tesla and the Great Pyramids is also not supported by relevant literature. 

In another example, a video discussing issues in green energy sources presented a misleading claim regard- 
ing the denial of climate change: 


Green energy also has a significant impact on price growth. Its tariff is the most expensive, but the rules of the energy 
market oblige us to buy green energy in full. We are being persuaded that the transition to green energy will reduce 
CO, emissions and save the world. (...) But even if we cover the entire surface of the planet with solar panels and wind 
turbines, the energy they will produce won't be enough. Has it only now dawned upon us that the sun doesn’t shine 
at night, and the wind doesn't blow all the time? In such conditions, power outages are unavoidable. (...) Wouldn't it 
be easier for us to move straight into a cave, or better yet, straight into a grave so as not to emit extra CO,? Which, by 
the way, has nothing to do with climate change. (V16, audio transcription) 


Based on this example, several noteworthy claims emerge that need further scrutiny. First, the argument that 
green energy is more expensive than fossil fuel energy is only partially true. Formulating a strong claim necessitates 
consideration of various analyses, yet the video fails to provide evidence or analyses supporting this argument. 
Secondly, the claim that wind and solar energy sources are not reliable is also partially true. To support such a claim, 
it is essential to present evidence, studies, and an explanation of the factors considered, which is notably absent in 
the example from the video. Lastly, the claim that green energy sources are not reliable in extreme weather condi- 
tions is also not completely true. Different energy sources can suffer damage due to weather conditions. Based 
on these considerations, notably, analyzing these energy sources and their costs, tariffs, and performances under 
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various conditions is difficult; however, the claims made by the author of the video do not seemingly consider all 
other variables and can, thus, be viewed as misleading. 

Another claim made in V16 is regarding the influence of CO, on climate change. This kind of denial of the 
impact of CO, on the climate change is also not well supported by relevant literature. In the video, this argument 
is merely presented without substantial argumentation to validate the claim. Scientific discourse supports the 
understanding that increased levels of CO, contribute to global warming and climate change. The lack of in-depth 
analysis or citation of credible sources weakens the credibility of this particular claim. 

Another video on energy sources proposes the existence of an engine called a fuel-free generator (FFG) that 
produces less pollutant energy. 


Thus, the fuel-free generator (FFG) is not a perpetual motion machine since it requires an input of energy, but a 
smaller amount than what energy it can produce. An obvious question arises: how, then, will energy be generated 
if no fuel is required? That we know from the school physics class that according to the law of energy conservation 
we can't get more network output than we had spent on it. But this law only works in closed systems. Whereas FFG 
is an open system, and it receives energy from the surrounding space, scientists of the Russian Academy of Sciences 
experimentally confirmed that, for example, during the operation of magnetic FFG, the space around the generator 
does not heat up but cools down instead. (V27, audio transcription) 


The claim that FFG is not a perpetual machine is false because FFG, as it is described in the video, continues 
the movement once set in movement, which is basically the definition of a perpetual motion machine. Second, 
the claim that according to physics and the law of energy conservation, “we can’t get more network output than 
we had spent on it” (V27), which is made in the video, is also inaccurate. The law of energy conservation states that 
energy will be conserved, that is, it cannot be created or destroyed. According to the law of thermodynamics, the 
output energy of a system cannot be higher than the input; in fact, no engine is capable of manufacturing 100% 
of the input energy into energy output. It is also important to emphasize that the law of energy conservation ap- 
plies to open and closed systems, not only to closed systems, as is claimed in the video. The differences between 
these types of systems rely on the transfer of energy and matter from the system to its surroundings. Regarding 
the claim that space around the machine becomes cooler, the idea that heat is transferred from a warmer to a 
cooler body until both achieve equilibrium is well known. Becoming cooler for the surrounding of the machine 
indicates that the surrounding is transferring heat to the machine, which seems very unlikely. A working machine 
would naturally generate heat due to movement, such that having a lower temperature than its surroundings is 
very unlikely for the machine. 

Lastly, another claim identified in this study is related to dark matter as a source of energy as stated in the 
following example: 


We just discussed how dark matter and dark energy are literally an energy source that is like 0.0. We can figure out 
how to harness this energy. We would literally be able to do anything. The aliens have been trying to help us and 
trying to get us to understand that we don't need to be so greedy about things like oil and all this. If we were using 
dark matter for what it could be used for, then we would be able to do anything and everything that the aliens do. 
In order to solve the dark energy equation, you have to be able to use at least 3% of your brain. It said most people 
only can use 1 to 2% if you're lucky. (V17, audio transcription) 


In relation to dark matter as an energy source, there is no evidence to support this claim. Regarding the idea 
of aliens talking is even more misleading compared with the other statements because no scientific evidence ex- 
ists of aliens contacting humans or attempting to teach humans. Another misleading aspect of the video that is 
not necessarily connected to environmental issues is the use of a percentage of the human brain, which has no 
scientific basis. 


Climate Change 
The study identified videos that presented denial of global warming and climate change. It found ideas, such 


as the Earth being cold most of the time; the use of cycles to explain why climate is changing; or even speeches 
blaming low-income countries for climate change. A few representative examples are presented below. 
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Speaker 1:1 think that the true data on Earth is that the Earth is cold most of the time, that right now, we should be 
grateful that it’s nice and cozy because we can live when it’s warm. But | think that the data might indicate that Earth 
is cold more often than it's hot. 

Speaker 2: Well, that’s what Randall Carlson has said and what Randall Carlson said really freaked me out. He says 
global warming is not scary. Those global cooling, that’s what's really scary. (V1 audio transcription) 


The first claim in this example is that Earth has been cold most of the time. However, this assertion lacks evidence 
in the relevant literature. Another claim made in the speech from V1 is that humans can live when the climate is 
warm, but evidence for this claim is also lacking. In human history, climate change has been identified as a reason 
for the collapse of early human civilizations. To gain credibility, these claims need a more comprehensive examina- 
tion of historical and scientific literature. Analyzing documented cases of civilizations affected by climate change 
and presenting empirical evidence would contribute to a more nuanced understanding of the intricate relationship 
between climate and human civilization. The absence of such evidence in the video weakens the persuasiveness 
of the claims, highlighting the need for a more thorough and well-supported discussion on these complex topics. 


In another example, the idea of cycles appears in one of the speeches from the video, which summarizes 
diverse parts of the speeches. The video does not seem to intend to spread misinformation; however, the creators 
added a few misleading claims combined with scenes of animals on Earth, such as polar bears and penguins, and 
asad song in the background. Nevertheless, misleading claims have been presented: 


Speaker 1: Oh, the world goes through cycles. 
Speaker 2: Oh, what? You mean the world warmed up without human intervention? (V4 audio transcription) 


This video claims that Earth undergoes cycles, suggesting that it warmed up without human intervention. 
While it is true that certain elements affect the climate in cyclic periods, there is no evidence to support these ele- 
ments being solely responsible for the dramatic changes our planet has been experiencing. The video introduces 
this concept without a discussion or supporting evidence, revealing a misleading nature. This misleading aspect 
arises from presenting information without credible sources or in-depth discussion, potentially leading viewers 
to misunderstand the intricacies of the topic. 

Another example related to the topic of climate change is the responsibility of and action on climate issues. 
The same speech appeared in two videos; the first presents only the full speech; the second reacts to that speech 
in which the person reacting agrees: 


Let us all accept right here, right now, that we are living through a climate emergency and our stocks of polar bears 
are running extremely low. | join you in this view. | truly do. Now what are we to do about this huge problem facing 
humanity? What can we in Britain do? We can only do one thing. Do you know why? This country is responsible for 2% 
of global carbon emissions, which means that if Britain was to sink into the sea right now, it would make absolutely no 
difference to the issue of climate change. You know why, cause the future of the climate is going to be decided in Asia 
and in Latin America by poor people who couldn't give a shit about saving the planet. (...) It’s going to be decided by 
poor people in Asia and Latin America who don’t care about saving the planet. Do you know why? Because they're 
poor. (V5 and V8, audio transcription) 


Regarding the claims made in these videos about the responsibility of the United Kingdom on climate change, 
it is important to note that the claims are not taking a broader approach to making such conclusions. The video 
claims that Britain is responsible for 2% of global carbon emissions. However, when we examine the data for 
Brazil and Mexico, the only two Latin American countries with emissions higher than the UK, their values are also 
relatively low. It is crucial to note that total emissions alone are insufficient for drawing conclusions. For instance, 
when looking at the same data for some Asian countries with higher CO2 emissions, such as China and India, they 
have large populations. China, the United States of America, and India occupy top positions in the ranking, but 
their large populations contribute to higher total emissions. Examining the data further, the emissions per capita 
of the United Kingdom are higher than those of Brazil. Therefore, considering only total emissions is inadequate 
for the conclusions made in the video; countries with larger populations will naturally have higher total emissions. 
Hence, considering various variables is necessary, an aspect seemingly overlooked by the video. 
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Pollution 


The study identified four videos under this topic, which were related to a chemical spill that occurred in the 
United States due to a train derailment. The videos refer to the East Palestine train derailment that occurred on 
February 23, 2023. The videos about the accident are considered misleading because they claimed that the ac- 
cident was intentional: 


| see a pattern here, and | have a prediction. Now, some of you might call me absolutely insane and, boy, | do hope 
that | am. | hope that | am wrong. But if you see me, if you're seeing me, and we're on the same channels, there's a 
good chance that you're woke and you're aware that American history is ripe with horrible atrocities and genocide. | 
don't think this is an accident. | don’t think that all of the poison going into the water in Ohio, in Philadelphia, now in 
Montana, there was another one recently in Hawaii, all of their aquifers are being polluted. (V29, audio transcription) 


In this first example, the content creator claims the intentionality of the train derailments; however, the fact 
that the government and corporations caused it intentionally lacks a scientific basis. The videos have a conspira- 
tory nature, which are centered on disbelief in the news and the government; for this reason, they are considered 
misleading videos. 

The study identified another video that fits into this topic: 


I now have to become a farmer because, well, all of our food is poisoned. And | also have to become an herbal medicine 
doctor because, like, our doctors are just prescribing poison. And | now bathe in apple cider vinegar, because, well, 
everything | was using on my body was poison. Oh, and laundry detergent. Yeah, now | use baking soda and vinegar, 
because, well, laundry detergent and fabric softener are poison. And oh, and bottled water is like a fake thing. It’s not 
really good for you or better for you. Also, it might be a little bit better for you than tap water because, well, that’s full 
of poison. | don’t believe any fucking doctor, any politician. | cannot trust the news. | have to be a fucking PhD to dig 
and find the truth in everything. (V12, audio transcription) 


In general, the author of the excerpt expresses disbelief in various sectors of society. The video has a conspira- 
torial nature; for this reason, it is considered misleading. Regarding the idea of food being poisoned, the creator 
does not specify any particular ideas, but the video likely refers to pesticides, which are considered problematic 
for human health and the environment. However, previous studies have demonstrated that people who apply 
pesticides are at actual risk. Making an assumption about poisoning from food to consumers is relatively more 
complicated. Governments implement regulations for the use of pesticides in food. Thus, further studies on the 
consequences and effects of the use of pesticides in food for consumers are required before making a strong claim 
that ‘all food is poisoned’ 

Other claims made in the video refer to the idea that doctors are prescribing poisoning to their patients; 
using soap is bad for the body, problems regarding laundry detergent and softeners, and bottled water and tap 
water being poisoned. These claims lack scientific evidence. The argument that using soap is bad for the body, for 
example, has no scientific basis. The prolonged use of soap may be problematic for the skin, but its normal use is 
not considered a significant problem. Regarding laundry detergent and softeners, these products can indeed cause 
more serious environmental problems. The claim that bottled water and tap water are poisoned is also conspirato- 
rial; there is no credible source cited in the video to support this argument, and scientific evidence points to their 
general safety, as will be discussed further. 

Another video was related to nuclear waste, which led to a mutation in a fish in Argentina: 


How did The Simpsons know this was going to happen? In an episode of The Simpsons that aired in the 1980s, Lord 
Simpson catches a three-eyed fish from the waters near a nuclear plant. This fish is named Blinky, and later in the 
episode, it’s revealed that the nuclear waste dumped in the water from the plant was responsible for the creation of 
this fish. Fast forward to 2017. An actual three-eyed fish was found in the waters of Argentina near a nuclear plant, 
and scientists determined that the cause of the mutation was the nuclear waste in the waters. Well, it might seem 
like a coincidence, but. (V18, audio transcription) 
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This study discovered several news articles related to the case, dated as far back as 2011, and identified the 
use of the same picture as that in the video. The incorrect timeline already suggests that the information may 
be misleading. In summary, the absence of trustworthy news reports on the case contributes to classifying it as 
potentially misleading information. 


Environmental Degradation 
Under this topic, one video was found: 
Speaker 1 (Moon): Hey, yo Earth, why you look so sad? You the only rock with intelligent life on it in this solar system. 


Be happy like the sun over there. 
Speaker 2 (Sun): I’m going to explode and kill everything | know and love in 200 billion years. 


Speaker 1 (Moon): Wow. Too bright. Too bright. 

Speaker 3 (Earth): I'm sad because | just found out I’m dying. 

Speaker 1 (Moon): Lying by what? 

Speaker 3 (Earth): No, dying. My people are slowly killing me. 

Speaker 1 (Moon): Oh shit. 

Speaker 3 (Earth): Yeah. 

Speaker 1 (Moon): Don’t they need you to, like, breathe and live and scroll through social media on their iPhones? 
Speaker 3 (Earth): That’s what | thought. 


Speaker 1 (Moon): We gonna get the sun to burn they assess. Just tell them what happened to the people on Mars. 
(V28, audio transcription) 


V28 contains two instances of misleading information. The first, though not directly related to environmental 
issues, is noteworthy. It falsely claims that the sun will explode in 200 billion years, which is incorrect. The second 
involves the mention of people on Mars, lacking scientific evidence. 


Biodiversity 


The videos under this topic were related to animals who were rated by their appearance. However, the audio 
is no longer available in one of the videos due to sensitive content, which may be an indicator that people on the 
platform reported the video. Nevertheless, the video has subtitles: 


Hello, and today we'll be rating different species of armadillo. First up, we have the pink fairy armadillo 1,000 out of 
10, partially because it’s the smallest armadillo species, but also partially just because it doesn’t look real. It looks 
like sentient sushi, and | really vibe with that. Next up, we have the giant armadillo — 15 out of 10. If | saw this in the 
forest, | would run. | understand that it’s not trying to eat me, but if you look at its claws, | think it successfully could 
eat me. (V9, audio transcription) 


The example shows the rating for armadillo species, which is the intent of the full video — rating different 
species. Initially, the video is seemingly harmless, but it can have consequences for the conservation of species. 
Speeches that emphasize the esthetic traits of animals can be misleading and lead to consequences for the future 
of the species. Another video that rated fish appearance seemingly attempted to emphasize these fish as endan- 
gered species, but rating their appearances is not beneficial for the future of the species. 


Discussion 
Energy Source 
Based on our results, it can be seen that there was a diversity of misleading claims related to the topic of Energy 
sources. First, the claim of V20 is about Pyramids as an energy source and the association of Nikola Tesla’s works 


with this pseudo-theory. There was no mainstream archaeological or scientific evidence found in relevant books 
such as Brier and Houdin (2008), Lehner (1997), Shaw (2003), Smith (2018) and Wilkinson (2003) that supports the 
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claim that the Great Pyramid was not a tomb. For example, Brier and Houdin (2008) explicitly mentioned that the 
purpose of the Great Pyramid of Giza was to “house the body of the dead king” (p. 14). Regarding the association 
of the works of Nikola Tesla and the Great Pyramids, no association was found in books about the scientist’s life 
and work, such as Cheney (2001) and Seifer (1996). 

Through the example of V16, some other misleading claims emerged, as can be seen from the results. The 
first one is about the expensiveness and reliability of renewable energy sources. Examining reports, renewable 
energy sources have become cheaper overtime. In fact, according to a report by the International Renewable 
Energy Agency (2023) in 2022, “solar PV's global weighted-average LCOE is 29% lower than the cheapest fossil fuel- 
fired option” (p. 17). Second, about the reliability of wind and solar energy sources. Tong et al. (2021) analyzed the 
capability of solar and wind as energy sources to meet electricity demand in 42 countries and demonstrated that 
these energy sources can be considered reliable. The authors reported that solar and wind energy sources would 
be able to satisfy around 72%-91% of hours of energy for the 42 countries in a scenario that does not consider the 
storage of energy. In a context considering the storage of energy, with 12 h of storage, the results go to 83-94% 
of hours being satisfied with the use of solar and wind energy sources for the 42 countries. Nevertheless, even 
for high-performance systems, unmet demands may occur annually for hundreds of hours. Regarding extreme 
weather, different types of energy sources are susceptible to damage in extreme weather conditions, as pointed 
out by Zamuda et al. (2013) in a report by the United States Department of Energy. 

In V16, a claim regarding the denial of CO2’s influence on climate change could also be found. As discussed 
by Oreskes (2018), the relationship between CO, and climate change is widely accepted. For example, the author 
provides extensive references to reports from respected scientific institutions such as the Intergovernmental Panel 
on Climate Change, the National Academy of Sciences, the American Meteorological Society, the American Geo- 
physical Union, and the American Association for the Advancement of Science. Moreover, the author emphasized 
that in an analysis of 928 abstracts of papers published from 1993 to 2003, none of them presented data to refute 
the consensus position on global climate change, which is defined by the author as the human influence on climate 
change and consequently the influence of CO, on global climate. 

Another misleading claim found in the topic of energy sources is related to free, non-pollutant energy sources 
as the free-fuel generator (FFG) presented in the example of V27. In the example, it is said as if this generator is not 
a perpetual machine. However, in the literature, the definition of a perpetual motion machine, according to Tsaousis 
(2008), a perpetual machine is, “a machine which, since set in function, continues to function perpetually without 
supplying any energy” (p. 53). Regarding the difference between closed and open systems, Kittel and Kroemer 
(1969) explained that a closed system can transfer energy but not matter, whereas an open system can transfer 
both to its surroundings. This is exactly what is being described as an FFG in the video, and, in this sense, the claim 
is misleading. A perpetual motion machine cannot work based on simple thermodynamics laws. 

Lastly, V27 presented claims regarding dark matter being an energy source. In the mainstream literature about 
dark matter, there is no support for these claims. For example, according to the definition by Ryden (2017) ina book 
entitled Introduction to Cosmology, dark matter can be defined as “any massive component of the universe which 
doesn't emit, absorb, or scatter light at all” (p. 27). Weinberg (2008) explained the nature of dark matter as, “most 
likely it is made of exotic particles, which do not interact electromagnetically and hence do not produce light” (p. 
84). Another misleading claim presented in this video is about aliens trying to contact us. Regarding the argument 
of aliens talking and teaching humans, according to Talbert (2003), this has no scientific evidence. Also, the claim 
about how much percentage of the brain is used, according to Herculano-Houzel (2009) and Zjacic (2023), has 
become a common myth in the past century but has no scientific basis. 


Climate Change 


As shown in the results, the topic of climate change also presented a significant number of misleading claims. 
The first video on this topic (V1) presented a dialogue that claimed that Earth is cold most of the time. Firstly, an 
analysis by Scott and Lindsey (2023) includes a graphic demonstrating the estimated temperature of Earth in the 
last 500 million years. Moreover, the graphic indicates that global surface temperatures have been mainly warm. 
Therefore, no evidence exists to support the claim. Second, the claim that humans can live when the climate is warm 
is discussed by Ruddiman (2014) in a book entitled ‘Earth’s Climate: Past and Future; which explores the collapse 
of the Mayan civilization. The author cites evidence that the collapse was due to drought, as this period coincides 
with a higher amount of lake sediments, an indicator of drought. Despite the evidence pointed out by Ruddiman 
(2014), designating climate change as a unique cause of changes in early civilizations is difficult. 
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V4 presented another misleading claim related to Earth working on cycles, and for that reason, it warmed 
up without human intervention. Regarding the cycles that affect climate on Earth, Ruddiman (2014) in his book 
explained that tectonic, orbital, or millennial time scales are unable to explain the recent increase in global tem- 
perature. The researcher also explained that short-term forcings, such as volcanic explosions or El Nino events, 
cannot account for the changes in global temperature. Alternatively, approximately 10% of the total increase in 
temperature could be related to solar irradiance. In summary, the majority of changes in global temperatures can 
be attributed to human activities (Ruddiman, 2014). 

The claims presented in the videos V5 and V8, are related to the responsibility and action on climate change. 
According to the speech presented, the United Kingdom has little responsibility because the amount of emissions 
from the country is very small ona global scale. According to data from a report by the International Energy Agency 
(2022), sourced from the Emissions Database for Global Atmospheric Research, the United Kingdom accounted for 
335.36 Mt (0.89%) of global carbon emissions in 2021, ranking 17th among countries worldwide. The only Latin 
American countries emitting more total CO2 than the United Kingdom are Brazil, with a total of 489.86 Mt (1.29%), 
and Mexico, with 418.35 Mt (1.11%). When comparing emissions per capita, the United Kingdom and Brazil present 
totals of 4.95 and 2.28 t CO2/cap, respectively. Additionally, if you take into account the respective populations 
of nations, the United Kingdom is more polluting than Latin American countries measured by CO, emissions per 
capita. Climate change is a global problem and demands cooperation from different countries. Trying to take 
responsibility for data that does not take all variables into consideration is misleading. 

The V5 and V8 also argue that “poor” countries are responsible for climate change on Earth. However, data 
from Ritchie et al. (2020) reveals that high- and upper-middle-income countries are responsible for 86% of global 
emissions, indicating that poverty is not necessarily an aggravator of CO2 emissions. Moreover, developing coun- 
tries are frequently more vulnerable and affected by the consequences of climate change (Abeygunawardena 
et al., 2004). Other studies on the impact of climate change and poverty consider low-income populations more 
vulnerable (Hallegatte & Rozenberg, 2017; Hallegatte et al., 2018; Marotzke et al., 2020). 

One might argue that the video intends to emphasize that developing countries have the urgency to grow. 
However, stopping their growth is impossible, as per this excerpt: ‘You're not going to get them to stay poor’ (V5). 
Consequently, they will pollute and become the main agents responsible for climate change in the future through 
their growth. However, this argument overlooks the fact that developed countries will also continue to grow and, 
hence, contribute to climate change. According to the economic model that governs the world, the need for con- 
tinued growth will persist, with developed countries also working toward growth. 


Pollution 


From the results, it can be seen that misleading claims on the topic of pollution were related to conspiracies. 
V29 presented a conspiracy related to the chemical spill from a train derailment in the United States. In V12, also 
elements of conspiracy can be identified, those which are related to disbelief in different sectors of our society. 
These kinds of speeches have no scientific evidence. 

Specifically in V12, the claim was made regarding food poisoning. Probably related to the use of pesticides. 
According to Rani et al. (2021), the higher risk for people concerning pesticides is for those in direct contact with 
them, indicating that farmers and agriculture generally suffer more from pesticide poisoning. As mentioned, gov- 
ernments generally have regulations for the use of pesticides; for instance, according to data from Granby et al. 
(2019), violations of the safe levels of pesticide use were detected in 1.9% of domestic fruits and 6.7% of imported 
vegetables in the United States in 2003. In other words, a low percentage of food violates the country’s regulations. 
Another example presented by Carvalho (2017) points out that 97% of the 87,000 food samples of the European 
Union analyzed in 2014 were within legal limits; out of those, 53.6% did not present quantifiable residues, and 
43.4% had residues within regulations. 

Other misleading claims identified in V12 were about doctors prescribing poison to their patients. Additional 
misleading claims were related to soaps, laundry detergents and softeners being bad. In the literature, arguments 
related to medicine are emphasized by Boudolas et al. (2017), highlighting that advances in medicine have sig- 
nificantly contributed to the increase in human life expectancy. Regarding soap, Wolf et al. (2001) state that the 
negative effects of soap on the skin are occasionally exaggerated, and ‘with ordinary use for general personal 
hygiene, the majority of consumers will tolerate any kind of soap without any harm’ (p. 396). When it comes to 
laundry detergent and softeners, various researchers have pointed out the negative impact of these products, 
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specifically due to their ingredients and future disposal, which may contaminate water resources (Bajpai & Tyagi, 
2007; Gupta & Sekhri, 2014; Pettersson et al., 2000; Sabharwal, 2015). 

Lastly, V12 made a misleading claim regarding water being poisoned. Research on bottled water has reported 
little to no concern about its safety, and the same is true for tap water. Napier and Kodner (2008) emphasized that 
access to the full content of water quality is difficult, but bottled water is generally considered safe. Jain et al. (2019) 
also expressed that bottled water is generally safe to consume. According to the authors, tap water is even healthier 
and more environmentally sustainable, but evidently, it should be boiled and/or filtered before consumption. Raj 
(2005) conducted a study on stored bottled water and demonstrated that bottled water stored for a long time 
produced more bacterial growth over time than tap water did. 

V18 also made a misleading claim related to pollution. V18 presented a report about fish mutation due to 
nuclear waste. As mentioned in the results, the wrong timeline presented in the video for the report and the lack 
of credible news on the case already indicate the video as misleading. In the literature, various studies explore the 
effects of nuclear waste, radiation, and other types of waste on animals, including fish (Hoshina et al., 2008; Kolar 
& Gugleta, 2019; Lourenco et al., 2017; Souza & Fontanelli, 2006). However, no reliable report exists regarding the 
case of the three-eyed fish in Argentina. 


Environmental Degradation 


V28 presented a dialogue between Earth and the Moon. In this dialogue, some misleading claims are pre- 
sented. First, the wrong statement is that the Sun will explode in about 200 billion years. In fact, this will occur in 
about 5 billion years due to the evolution of the Sun into a red giant star. Secondly, the idea of Mars being home 
to people in the past. The video presented the idea that people died on Mars by not taking care of the planet as 
we are doing to Earth now, but this claim has no scientific evidence. 

The relevant literature sheds light on these issues. According to Frazier (2019) and the National Aeronautics 
and Space Administration (n.d.), in approximately 5 billion years, the sun will evolve into a red giant star, expanding 
to 200 times its current radius. Regarding the presence of people on Mars, as explained by Furfaro (2023), there is 
evidence supporting the past existence of liquid water and a thicker atmosphere on Mars. However, these facts 
do not prove that Mars was a home for people as stated in the video. 


Biodiversity 


The example provided by V20 presented a video where armadillo species were rated based on their appear- 
ances. This content can also be considered misleading due to its consequences and influence. Research on the 
conservation of animals demonstrated that differences exist in the process of conservation, support, and attention 
for diverse species, based on human preferences for specific aesthetic traits (Papworth & Curtin, 2022; Pinho et al., 
2014). For example, animals that are considered cute or charismatic frequently receive more media attention and 
more conservation support if compared with animals that are considered ugly or less charismatic. Colléony et al. 
(2017) and Martin-Ldpez et al. (2007) also found that people are willing to donate to more charismatic animals 
regardless of ecological or scientific considerations. Small (2012) highlighted that considering not only the aesthetic 
or economic value of a species but also the diversity and vital ecological roles they play in sustaining the necessary 
ecosystems for human survival is important. 


Conclusions and Implications 


This research aimed to comprehend the dissemination of misleading information concerning environmental 
issues through TikTok videos. In contrast to prior literature primarily focused on climate change, this investigation 
broadened its scope to explore a wider range of environmental issues. The analysis of results and subsequent 
discussions led to insights into the nature of misleading content within the videos. The findings revealed that 
TikTok videos with misleading information predominantly centered on topics such as energy sources, climate 
change, pollution, biodiversity, and environmental degradation. Furthermore, among the diverse array of mis- 
leading claims, topics such as non-pollutant energy sources (e.g., the Great Pyramids as power plants and Nikola 
Tesla) and pollution-related conspiracies expressing distrust in government, companies, science, or news were 
frequently mentioned. 
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This research concludes that, unlike previous literature that predominantly focused on climate change denial, 
the diversity of topics targeted by online misleading information on environmental issues extends beyond just 
climate change. The range of topics addressed underscores the pervasive impact of online misleading information 
across various facets of environmental discourse. This suggests the need for educators and education systems to 
broaden their focus on specific topics to effectively address the current issue of online misinformation. 

The identification of specific topics and misleading claims in this research serves as a valuable resource for 
science educators dealing with these subjects. Grounded in relevant literature, the specificities identified in this 
research can serve as exemplary resources for educators aiming to address misleading claims related to environ- 
mental issues in their classrooms. The findings also provide insights for enhancing science education curricula, 
emphasizing the necessity of a multidisciplinary approach in addressing environmental issues and combating 
misinformation. 

Within the predominant monodisciplinary educational structure, successful implementation of a multidis- 
ciplinary strategy requires cooperation and strategic planning among educators. For example, science teachers 
from various domains can collaborate to identify common ground facilitating integrated exploration of online 
misleading information on environmental issues. Projects on climate change, involving contributions from phys- 
ics, chemistry, biology, and earth sciences, can help students analyze scientific data, explore societal impacts, 
and articulate their findings, incorporating specific skills like source evaluation, fact-checking techniques, and 
awareness of bias. 

In summary, this study, through content analysis, deepens understanding of topics often targeted by mis- 
leading information. It offers a detailed examination of video content, analyzing discourses, identifying specific 
misleading claims, and discussing each in the context of relevant literature. Future extensions could involve 
analyzing non-misleading videos to explore how science is utilized and studying misleading information across 
various scientific topics to comprehensively address this ongoing issue. Additionally, examining other platforms 
and comments on misleading videos can provide new insights into addressing misinformation in social media 
for science education. 
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Abstract. Building 21st-century life science 
skills requires educating participants 
according to STEM abilities. Therefore, 

this research aimed to examine the 
effectiveness and feasibility of the STEM 
ability assessment framework in the 
practical learning environment. The study 
uses STEM coffee preparation experiential 
activity with a Royal Belgian siphon pot 

to construct a learning environment 

in the classroom. The study also 

develops two assessment instruments, a 
knowledge concept questionnaire, and an 
entrepreneurial scientific thinking scale, to 
examine their effectiveness and feasibility 
in the STEM learning environment. The 
results of the content validity index reveal 
the value of good-grade literature for 

two questionnaires. Kendall's coefficient 
of concordance (w) of the four reviewers’ 
responses shows that the inter-rater 
reliability of the two questionnaires reaches 
a better level. The Chi-square test found 
that this STEM learning environment is 
feasible and effective and will help the 
participants assess their STEM abilities. 
The entrepreneurial scientific thinking for 
preparing beverages of life science is rich 
in viability and efficacy for instrument 
creation and assessment. Future research 
lengthened the extraction process while 
also improving consistency. Last but not 
least, more teaching practices and research 
designs are available. However, the goal 

is for learners’ STEM aptitude to increase 
practice depth. 

Keywords: effectiveness and feasibility, 
entrepreneurial scientific thinking, life 
science, Royal Belgian siphon pot, STEM 
education 
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THE FEASIBILITY AND 
EFFECTIVENESS OF LIFE 
SCIENCE IN STEM PRACTICAL 
LEARNING ENVIRONMENT 


King-Dow Su 


Introduction 


STEM is the abbreviation of science, technology, engineering, and math- 
ematics. Tsupros et al. (2009) defined STEM education and believed that 
STEM education is an interdisciplinary educational approach that combines 
rigorous academic concepts with real-world courses, prompting students to 
integrate science. Researchers (Mizell & Brown, 2016) accepted this defini- 
tion by knowledge in STEM applied to link schools, communities, jobs, and 
businesses. Technology-integrated STEM education can arouse students’ 
willingness to learn more than other single subjects or technology-integrated 
subjects (Becker & Park, 2011). The United States led many countries to main- 
tain a competitive advantage in global economic development. Therefore, 
educators and policy-making departments should actively promote science, 
technology, engineering, and mathematics domain knowledge to take action 
in STEM regard (Kelly & Knowles, 2016). In science, both scientific thinking and 
scientific concepts construct the fundamental background of mathematics; 
in engineering and technology, which rely upon the application between 
mathematics and science; in mathematics, it helps learners integrate their 
understanding of scientific concepts. This interdisciplinary approach to STEM 
teaching is inseparable from the nature of science, technology, engineering, 
and mathematics. They have a unified structure throughout history, overlap- 
ping and merging (Broggy et al., 2017). 

The successful promotion of STEM fields contributes to the competitive 
advantage of the labour market and the stable development of the national 
economy (Bicer et al., 2017). Accordingly, STEM education is welcomed by 
more and more teachers, researchers, administrators, and policymakers, 
making it a popular interdisciplinary focus of attention (Brown et al., 2011). 
STEM education aims to develop learners’ critical thinking skills and enable 
them to become effective problem solvers on STEM-related issues (White, 
2014). Therefore, assessing STEM abilities will help learners choose a decision- 
making reference for future careers related to STEM (Enderson & Ritz, 2016). 
Saxton et al. (2014) pointed out that the assessment framework will facilitate 
the development of more effective tools for assessing learners’ STEM com- 
petencies. The study also pointed to a comprehensive STEM assessment 
framework that reflects the diverse structure of STEM education, including 
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a focus on experiential context and higher-order thinking skills in STEM-related content. However, the thinking 
skill cognition that forms the basis of the STEM assessment framework is reasoning ability (Mullis & Martin, 2017, p. 
22) and non-routine problems. Milgram (2007) pointed out that non-routine problems require students to activate 
innovations to integrate their reasoning abilities and find solutions. So far, the literature for the life science and 
technology disciplines has rarely examined the evaluation application of STEM interdisciplinary. 


Entrepreneurial Scientific Thinking in STEM Education 


In 2017, the U.S. Department of Education emphasized that integrated interdisciplinary STEM education can 
cultivate learners’ ability to understand information, make precise decisions, solve problems, and overcome future 
challenges. In the 21st century society, this importance is even more significant (Salmi et al., 2021; Struyf et al., 
2019). STEM education emphasizes knowledge application and practical experience to provide participants with 
multi-dimensional learning, such as knowledge, communication, skills, coordination, thinking, creation, design, 
and cooperation (Bybee, 2010; Moore et al., 2015; Tsai et al., 2021). Try to link schools, communities, work, and 
enterprises (Tsupros et al., 2009; Mizell & Brown, 2016) to integrate academic concepts and real-world courses. 
To show the core value of STEM Education through hands-on learning, brain thinking, and communicating with 
verbs (Baran et al., 2016; Tsai et al., 2018). 

STEM education emphasizes technology and scientific thinking to solve problems and connects the knowledge 
learned with life experience in practice (Mohtar et al., 2019; York et al., 2019). It is an interdisciplinary integration 
model of high-level thinking. This interdisciplinary model, Buang et al. (2009) theoretical model of entrepreneur- 
ial scientific thinking, and Kolb’s (1984) four-stage experiential learning all emphasize their actual participation, 
critical thinking, innovation, and problem-solving. Gunawan and Shieh’s research (2020) found that integrating 
entrepreneurial scientific thinking into STEM courses will impact students’ learning. Eltanahy et al. (2020) pointed 
out that applying entrepreneurial scientific thinking in STEM learning can improve learners’ design awareness and 
increase their ability to judge product value. Anmad and Siew (2021) also believe that entrepreneurial scientific 
thinking in STEM education will help cultivate talents who can solve problems, make decisions, and be innovative 
and creative to benefit the future society. As a result, STEM education supports the development of entrepreneurial 
scientific thinking skills in this study. 


The Importance of STEM Competency Assessments 


Shulman (2009) pointed out that assessment is a powerful tool to improve the quality of teaching. Ac- 
cordingly, the assessment of STEM abilities can help improve the quality of STEM education. Saxton et al. (2014) 
also believe that assessing STEM competencies in domain knowledge can promote the quality of learning and 
teaching. The evaluation methods include a formative and summative evaluation. In contrast to summative assess- 
ment, STEM evaluation is a formation, according to Capraro and Corlu (2013). Some scholars believe that formative 
assessment can involve timely feedback in each process and be effective (Han et al., 2015; Haudek et al., 2011). 
Compared with the research on summative assessment of STEM ability, although it is more specific and clear, it is 
rarely mentioned in the literature. A significant problem is the absence of a STEM pedagogical assessment frame- 
work (Capraro & Capraro, 2013; Ing, 2014). 

Researchers (Jang, 2016; Loukomies et al., 2013; Salmi et al., 2021; Struyf et al., 2019) pointed out that raising 
STEM learning environmental design was necessary. It could improve learning interest, motivation, and twenty- 
first-century skills across the interdisciplinary. Harwell et al. (2015) thought that the STEM assessment framework 
would help to assess the instrument development and verification abilities of STEM. Bicer et al. (2017) believed 
that STEM assessment models are necessary to examine students’ STEM skills and gain a profound understanding 
of STEM fields in an interdisciplinary manner. Science, technology, engineering, and mathematics (STEM) practised 
learning environments will be effective and successful engagement. However, no comprehensive pedagogical as- 
sessment frameworks enable the creation of a STEM learning environment (Makela et al., 2022; Struyf et al., 2019). 

Accordingly, this study develops a summative STEM ability assessment method and proposes the develop- 
ment of the STEM knowledge concept questionnaire and the entrepreneurial scientific thinking scale. 
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Purpose and Research Questions 


It is crucial to educate participants based on STEM abilities to build 21st-century life science skills. Therefore, 
this research aimed to apply the STEM pedagogical assessment framework to attempt to construct a learning 
environment of technology application in STEM coffee-making experiential activities. To develop assessment 
instruments for knowledge concept questionnaires and entrepreneurial scientific thinking, trying to examine the 
learning environment in the process of STEM education. Test the effectiveness of students’ entrepreneurial scien- 
tific thinking ability and evaluate its feasibility to pass the test evidence of empirical research on the framework 
structure. The research questions explored in this study were as follows: 

1. What is the domain of the STEM pedagogical assessment framework? 

2. What are the content and face validity of the STEM ability assessment instruments? 

3. Whatis the inter-rater reliability of the STEM ability assessment instruments? 

4. Whatis the feasibility and effectiveness of using this assessment instrument to examine learners in the 
STEM practical learning environment? 


Research Methodology 
General Background 


This research builds a competence evaluation system in life science using the design thinking STEM learning 
paradigm and the Royal Belgian siphon pot. This assessment framework is meaningful for the creation, verification, 
and experience of STEM ability. The study developed two standardized research instruments, the STEM knowledge 
concept questionnaire, and the entrepreneurial scientific thinking scale, to collect data and learn from the STEM 
learning environment. Six experts were rated based on eight evaluation questions, with the average value serving 
as the content validity index (CVI). The study uses descriptive statistics, the harmony coefficient, and the Chi-square 
fitness test to investigate the reliability, validity, and fitness of two assessment questionnaires. The content dimen- 
sions of the STEM ability assessment framework contain and define each subject and associated sub-dimension of 
science, technology, engineering, and mathematics. This framework exemplifies the various integration of STEM 
education with life science. The practicality and efficacy of the STEM practical learning environment, along with 
brain thinking, hands-on activities, and verbal argument, may assist students in constructing mental models in 
STEM education and promote students to have a deeper idea of life science and profound comprehension during 
the 2022 academic year. 


Participants and Ethical Approval 


The participants of this research, from the K-12 and K-14 of the Department of Hospitality and Tourism of 
the University of Science and Technology in Taiwan, two recruiting classes and 70 students (41% males and 59% 
females) who took the coffee beverage preparation in life science course participated in the experiential activities. 
They are similar in age (18-20) and have just started to learn drink-making. Furthermore, six people are involved in 
the role of experts. They reviewed as long as the logic of the questionnaire structure, the rationality, and fluency of 
the content. Distribution of their expertise included two professors of education and curriculum communication, 
two technology education, and two science education. They consisted of three females and three males, aged 55 
to 62, with more than 20 years of teaching experience in six different science and technology universities. 
Students’ involvement in this experiential teaching method is voluntary, and after completing the informed 
permission form, they gain practical experience in the STEM learning environment. After finishing the STEM cur- 
riculum, they fill out the questionnaire anonymously and code using Arabic numbers and English on ethical issues. 
Taiwan Ethics Committee, certificate number NCCU-REC-202205-E022, issued consent. 


Instrument Design 


The STEM ability assessment instruments included the STEM knowledge concept questionnaire and the 
entrepreneurial scientific thinking scale. As for the STEM knowledge concept questionnaire, divide the question- 
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naire with open-ended questions into three parts. The first part, the opening words, let voluntary participants 
understand that this questionnaire had nothing to do with the grades of the courses they have taken and only 
express their learning perceptions; the second part, there were demographic questions; in the final part, refer to 
Bloom's revised (Anderson & Krathwohl, 2001) cognitive taxonomy for open-ended test questions and design, 
allowing students to complete the test after the STEM experiential learning activities and present their revised 
Bloom learning outcomes and cognitive domain distributions in a cognitive taxonomy. The first draft has six aspects 
from remember, understand, apply, analyse, evaluate to create, and has three test items in each one. The first draft 
of the test questions will be adapted and revised about the learning outcome scale framework of Dedeturk et al. 
(2021), Honey et al. (2014), and Su (2019). To design a total of 18 questions about STEM knowledge concepts. The 
first draft invited the above six experts to conduct a substantive review and revise the content and logic of this 
questionnaire. Furthermore, eliminate illogical topics or contents according to the opinions and suggestions of the 
experts after the triangular correction to ensure the content and face validity of the questionnaire. 

Additionally, the degree of success is scored from 1 to 5 points by the six experts asked to participate in this 
study, using the 8 test questions from the literature (Burton & Mazerolle, 2011; BSNP, 2016; Wahono & Chang, 2019) 
as the scoring criteria. Item 1, the tool system preparation; Item 2, the clarity and readability of each item in the 
phrase; Item 3, the coherence between item and item; and Item 4, depending on the goal of tool creation Item 
integrity; Item 5, STEM knowledge, attitude, application, and accuracy of each item; Item 6, ease of use; Item 7, the 
survey results will tell the truth; Item 8, does not include race, ethnicity, religious issues, infringement of intellectual 
property rights, pornography, and prejudice (such as gender, region, etc.). Six experts scored according to these 
eight items and their average value as content validity index (CVI). This study also invites four of them to answer 
the content of the test items in the assessment instrument. The scoring standard refers to Gunter and Alpat (2017) 
to build reliability among raters and make it a formal STEM knowledge concept test questionnaire. 

There are 18 questions in the questionnaire, some of which are summarized as follows: Item 1 What is the rela- 
tionship between vapour pressure and boiling point when brewing coffee in a Belgian pot? (A) Directly proportional 
(B) Inversely proportional (C) Exponential relationship (D) Uncorrelated. Please write the reason:___ Ki ltem 2, What 
is the scientific principle of the seesaw when brewing coffee in a Belgian pot? (A) Siphon phenomenon (B) Cor- 
relation (C) Thermal radiation (D) Ideal gas. Please write the reason: ; Item 3, What is the scientific phenomenon 
of the Belgian pot brewing coffee? (A) Internal and external gravity (B) Internal and external mass (C) Internal and 
external volume (D) Internal and external pressure balance. Please write the reason: __. 

The initial draft of the test questions for the entrepreneurial scientific thinking scale corresponds to the design 
observation, new thinking, and innovation test questions of Schelfhout et al. (2016), Anmad and Siew (2021), and 
Bung et al. (2009). This scale comprises five creative and value aspects and 15 open-ended test items. The first draft 
requested the six experts stated above to evaluate and revise the questionnaire content and logic and to exclude 
illogical items based on the updated opinions and suggestions following the expert triangular revision to confirm 
the content validity. Process the CVI value of this scale in the same way as the above questionnaire. Simultaneously, 
four experts filled out the scale test questions to build the inter-rater reliability, resulting in a formal test question- 
naire for the entrepreneurial scientific thinking scale. The grading criteria for open-ended test questions depend 
on Ahmad and Siew (2021) and Ho et al. (2013) to understand students’ entrepreneurial scientific thinking abilities. 

There are 15 open-ended test items in the questionnaire on entrepreneurial scientific thinking, some of which 
are summarized as follows: Item 1, Explain that you have observed the advantages of using the Belgian siphon pot 
to brew coffee and the materials used: ; Item 2, Explain that you can watch the design used for brewing 
coffee using a Belgian siphon pot: ; Item 3, Explain that you can examine the rebreathing of coffee brewed 
in a Belgian siphon pot: 


Research Procedures 


The study approach consists of four steps: topic confirmation, assessment of STEM competence framework 
domains, development of STEM ability assessment instruments, and empirical research on the feasibility and ef- 
fectiveness. Figure 1 depicts the four stages of the 
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Figure 1 
Flowchart of the Research Process 


Results Statistical Analysis and 
Discussion 


Domains of STEM competency 
assessment framework 


research process. In Figure 1, the initial step is to analyse STEM literature to identify research issues. The second 
stage is the domains of the STEM ability evaluation framework. Following that, create a STEM ability evaluation 
instrument. Furthermore, when the participants have understood the experiential learning environment, have 
them fill out the questionnaire. This research finished statistical analysis, identified supporting literature from the 
analysis findings, and then composed the results into a report and published it. 


Data Analysis 


A questionnaire is an instrument designed in this study to collect quantitative data. The STEM knowledge 
concept questionnaire and the entrepreneurial scientific thinking scale are two parts of the STEM ability assess- 
ment. The purpose is to use statistical methods such as descriptive statistics, harmony coefficient, and Chi-square 
fitness test to analyse the reliability, validity, and fitness of the STEM ability assessment questionnaire. Implement 
all statistical approaches to use SPSS for MS Windows version 25 statistics. 


Research Results 
STEM Ability Assessment Framework 
The Royal Belgian siphon pot, created by the British shipbuilder James Napier (Bramah & Bramah, 1989), is 


utilized as the foundation for the context of science and technology education in this research. The image in Figure 
2 shows a hand-painted schematic and its elements. Use 
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The Royal Belgian Siphon Pot Hand Drawing and its Elements 
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Figure 2 as an educational technology to construct a STEM course learning model with design thinking. Figure 
3 depicts this STEM course learning approach and combines the functions of the siphon pot to design a learning 
mode for STEM courses with rich educational functions. It reveals the field of STEM ability assessment framework 
to construct learning domains in coffee preparation. It also includes the balance of the siphon principle, extrac- 
tion, and vapour pressure used in the science implementation process. The technology ability assessment includes 
metal heat transfer, control of gravity balance, and warm fire. Next is engineering capability assessment, including 
siphon installation, filter cloth installation, and alcohol lamp installation design. Lastly is the mathematical ability 
assessment, including raw material weight, temperature, water volume, alcohol volume addition, etc. In other 
words, building the STEM ability assessment framework with the Royal Belgian siphon pot is around mathematics 
in coffee preparation. The content dimensions of the STEM ability assessment framework include and clarify each 
theme and related sub-dimensions of science, technology, engineering, and mathematics. This content framework 
is indicative of the diverse integration of STEM education. 


Figure 3 
Domain of STEM Ability Assessment Framework 


Engineering 
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The accuracy of an evaluation instrument represented its validity. The measured data is more accurate when 
reducing the error. Six experts conducted a substantial review and logical modification of the content and logic of 
the two questionnaires in this study. According to the opinions and suggestions of experts after triangular correc- 
tion, modify or remove illogical items so that the two questionnaires meet the face validity of experts. Furthermore, 
experts accorded eight test items as score standards. Six experts scored the two questionnaires based on the eight 
items, and the descriptive statistics and average values after scoring are shown in Table 1. 


Content validity and face validity of research instruments 


Table 1 presents two expert content validity indices (CVI). They include the STEM knowledge concept ques- 
tionnaire and the entrepreneurial scientific thinking scale. In the STEM knowledge concept questionnaire, the 
distribution of each item ranges from .8333 to 1.0000, and the CVI of the mean score (.9458) and standard deviation 
(.0485). In the entrepreneurial scientific thinking scale, the distribution of each item ranges from .8667 to 1.0000, 
and the CVI of the mean score (.9542) and standard deviation (.0557). Table 1 shows the mean CVI for both research 
instruments, with indices above .94. 


Table 1 
Expert Content Validity and Face Validity of Research Instruments 


Code Content M SD 
QA STEM knowledge concept questionnaire 9458 0.0485 
QB Entrepreneurial scientific thinking scale 9542 0.0557 


Feasibility and Effectiveness of the STEM Practical Learning Environment 


The feasibility of a practical learning environment for STEM teaching, this study invited four interdisciplinary 
professors to answer the test questions of the research instruments, STEM knowledge concept questionnaire, and 
entrepreneurial scientific thinking scale. To assess the inter-rater reliability of the study instrument, we utilized 
Kendall’s consistency coefficient (w) based on the results of the four experts, as shown in Table 2. 

In Table 2, Kendall’s coefficient of concordance, w value of the STEM knowledge concept questionnaire is .718, 
reaching a significant level (p < .001). The other coefficient w value of the entrepreneurial scientific thinking scale 
is .664, a significant level (p< .001). Therefore, the feasibility of the STEM teaching practice learning environment 
is also verified. 

In addition, in terms of the effectiveness of the STEM teaching practice learning environment, the Chi-square 
test of the fitness of the STEM knowledge concept questionnaire (y?= 34.469, p< .001) showed a significant level; 
the Chi-square test of the fitness of the other entrepreneurial scientific thinking scale (y?= 29.888, p < .001) also 
showed same level, Table 2 showed the data. This Chi-square test of goodness-of-fit reveals that the research instru- 
ment is effective for students’ knowledge concepts and entrepreneurial scientific thinking in the STEM practical 
learning environment. 

To sum up the research results, the feasibility and effectiveness of the STEM practical learning environment, 
combined with brain thinking, hands-on and verbal debate, can help students develop mental models in STEM 
education and promote students to have a deeper concept and profound understanding of life science. 


Table 2 
Feasibility and Effectiveness of Two Research Instruments 


Code Content o x Dp 
QA STEM knowledge concept questionnaire 118 34.469 < .001 
QB Entrepreneurial scientific thinking scale 664 29.888 < .001 
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Discussion 


Previous research (K6nings et al., 2014) had shown that integrating student and teacher perspectives in expe- 
riential learning and designing for STEM environments could improve design quality. The dynamic domain of the 
STEM ability assessment framework included science, technology, engineering, and mathematics and integrated 
into coffee-making experiential activities in the Royal Belgian siphon pot. In the practical learning environments, 
the subdomains represented each main domain with the educational technology application. The learning environ- 
ments consisted of the siphon principle, extraction, and the balance of vapour pressure to implement in life science. 
The siphon pot included metal heat transfer, control of gravity balance, and a warm fire to carry out in technology. 
Engineering comprised siphon installation, filter cloth installation, and alcohol lamp installation design to fulfill. 
Mathematics incorporated raw material weight, temperature, water volume, and alcohol volume in addition to put- 
ting it into effect. Bicer et al. (2017) also believed a STEM assessment model is necessary for students’ STEM abilities. 

This study uses the Belgian Royal Coffee Pot in coffee beverage preparation in a life science course to solve 
STEM education real-world problems, such as dietary issues, art to relieve stress, etc. With the incorporation of 
vapour, gravity, pressure, and fire, brewing coffee is not only a romantic situation but also a piece of art. Integrat- 
ing these scientific principles into one, drinking coffee becomes a rational and emotional game. The learning in 
the course is full of the romantic sentiment of royalty, allowing students to feel the flip of the curriculum, which 
is different from traditional science course learning. In addition, apply the scientific principles of this equipment 
to STEM education in life science courses on hot beverages such as fruit tea, flower tea, hot matcha, hot milk tea, 
green tea, oolong tea, and black tea. 

According to Saxton et al. (2014), the STEM ability assessment framework incorporates the multifaceted 
framework of STEM education by emphasizing higher-order thinking abilities in the cognitive component and 
STEM-related coffee-making experiential activities in educational technology. STEM stands for science, technology, 
engineering, and math integration. In other words, the STEM learning environment blends mathematical opera- 
tions, engineering design, technology manipulation, and scientific cognition before using those concepts to solve 
STEM challenges. Through this study reported in the article, this research constructed the STEM ability assessment 
framework domain, as found in the study of Harwell et al. (2015), which will help the instrument development and 
verification of STEM ability assessment. 


Content Validity and Face Validity of Research Instruments 


Oluwatayo (2012) pointed out that face validity is the researcher's subjective comment on the presentation 
and relevance of the measurement instrument. That is, whether the items in the instrument appear relevant, rea- 
sonable, unambiguous, and distinct. Gelfand et al. (1975) stressed that face validity was the weakest validity form. 
Many people thought it was not the form of validity in the strictest sense. Therefore, Straub et al. (2004) proposed 
content validity as the degree to which items in an instrument reflect the content domain to which it will generalize 
(Taherdoost, 2016). Some researchers (Lewis et al., 1995; Boudreau et al., 2001) strongly recommend the application 
of content validity when developing new instruments. Content validity involves evaluating a new survey instrument 
to ensure it includes all necessary items and eliminates unrequired items in other structural fields. Accordingly, this 
study examined the content and face validity of research instruments and assured their quality and researchers’ 
ability to obtain accurate data before using them in actual research. 

This study used the expert CVI value and invited six experts to evaluate the above eight items of the second 
questionnaire as the expert's CVI value to establish content validity. In the STEM knowledge concept questionnaire, 
the CVI values of each item are above .8333, and the overall average is .9458; in the entrepreneurial scientific think- 
ing scale, the distribution range of the CVI values of each item is above .8667, and the overall average is .9542. The 
CVI values of the two questionnaires are better than .78 in the literature (Polit et al., 2007) and better than the CVI 
value of the new instrument suggested by Davis (1992), which should be greater than .80. 

To sum up, the CVI value of each item in the two questionnaires was better than the literature value. There- 
fore, the two questionnaires developed in this study had high content validity and face validity, which would help 
collection and improve the accuracy of the data in this study. Makela et al. (2022) found that the involvement of 
experts increases the creation of STEM learning environments and skills assessment. 
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Feasibility and Effectiveness of STEM Practical Learning Environment 


This research uses Kendall's coefficient of concordance and the Chi-square test of goodness-of-fit as statisti- 
cal approaches to examine the feasibility and effectiveness of evaluation in the STEM practical learning environ- 
ment. However, two research instruments, the STEM knowledge concept questionnaire and the entrepreneurial 
scientific thinking scale in STEM competency assessment play a role in the decision of learning environment. The 
concordance coefficients of the two research instruments are all between .6 - .8, indicating that the reviewers 
have a better degree of consistency between the two STEM competency assessment instruments (Marozzi, 2014; 
Su, 2019). Ozkan and Topsakal’s research (2021) pointed out that in learner-centred STEM education learning, the 
feasibility and effectiveness of the environment will help learners understand concepts and have a positive impact. 

This research can further understand the degree of inter-rater consistency among the four raters from the 
value of the concordance coefficient, which belongs to the better level, and constructs the inter-rater reliability. In 
addition, based on the Chi-square test of goodness-of-fit, two research instruments of the STEM knowledge con- 
cept and the entrepreneurial scientific thinking scale are significant. This research instrument presented effective 
characteristics in the entrepreneurial scientific thinking of students in STEM education. To sum up, the feasibility 
and effectiveness of evaluation in the STEM practical learning environment mean that the two instruments are 
feasible and effective to use in the STEM ability assessment. 


Value and Application of the Two Tools 


In summary, this study applies the scientific principles of the Royal Belgian siphon pot to develop a summa- 
tive STEM ability assessment tool, namely the STEM knowledge concept questionnaire and the entrepreneurship 
scientific thinking scale, to assess the learning effectiveness of students’ STEM cognitive levels and entrepreneurial 
scientific thinking skills, respectively. The value and application of the two instruments are as follows: 

According to Shulman (2009), assessment is a potent instrument for raising teaching effectiveness. Thus, STEM 
competencies may contribute to raising the standard of STEM instruction. In the light of Saxton et al. (2014), STEM 
skills that evaluate domain knowledge may also improve the calibre of learning. Based on this scientific education 
principle, this research developed two instruments for summative STEM ability assessment: the questionnaire of 
knowledge concept used Bloom's revised cognitive taxonomy (Anderson & Krathwohl, 2001) to conduct open-ended 
test question propositions and design. It assesses students’ learning efficacy on six levels: memory, understanding, 
application, analysis, evaluation, and creation. In terms of the entrepreneurship scientific thinking scale, use an 
open-ended questionnaire to design five aspects of entrepreneurial scientific thinking ability. Such as observation, 
fresh ideas, innovation, creativity, and value are examples of such skills. 


Scientific Education Principles and Insights 


The Royal Belgian siphon pot integrates the scientific principles of vapour, gravity, pressure, and fire to 
construct a STEM learning environment as a teaching practice site. According to Harwell et al. (2015), a learning 
environment will enhance STEM experiential learning independence in the design and evaluation. Bicer et al. 
(2017) also believe that a STEM experiential learning environment and STEM abilities for students are necessary. 
Based on this research, develop summative STEM ability assessment instruments, a STEM knowledge concept 
questionnaire, and an entrepreneurial scientific thinking scale. The conducted research study by Enderson and Ritz 
(2016) highlights that the creation process of these two tools yielded valuable insights that will aid learners in their 
decision-making to future engagement with STEM-related careers. Applying entrepreneurial scientific thinking to 
STEM education might enhance students’ comprehension of design and assess the worth of products (Eltanahy 
et al., 2020). Additionally, Ahmad and Siew (2021) thought that incorporating entrepreneurial scientific thinking 
into STEM education will assist in developing an aptitude for problem-solving, decision-making, innovation, and 
creativity to contribute to society in the future. 

The STEM experiential learning at the Royal Belgian Siphon Pot enriches their visions and innovative value in 
the domain knowledge of life sciences. It improves their effective and meaningful scientific cognition in a STEM 
practical context. It is a unique highlight of this study for the above STEM studies. 
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Conclusions and Implications 


This research described the development process of the application of life science and educational technology 
in the STEM practical learning environments. Create their learning settings during this procedure with five Royal 
Belgian siphon pots as instructional equipment. The design thinking supported STEM experience learning environ- 
ments and integrated the STEM ability assessment framework in the life science domain. Integrating student and 
teacher perspectives in experiential learning and designing for STEM environments could enhance design quality. 
The results evidenced that the framework would help the instrument development and verification of STEM ability 
assessment in this research. 

Based on the analysis of the CVI values of content and face validity, a high degree of validation is helpful 
for data collection in this research. The two research instruments of this article revealed that the two designed 
questionnaire research variables are correct and accurate and have high index values in terms of academicism, 
systematisms, objectivity, logic, and rigour. A high content and face validity would improve the quality of a new 
instrument. The involvement of experts increases the creation of STEM learning environments and skills assess- 
ments. The results examined the content and face validity of research instruments and assured their quality and 
researchers’ ability to obtain accurate data before using them in actual research. 

This research used Kendall's coefficients of concordance (w) of the results of the four reviewers to conduct 
inter-rater reliability. Based on the STEM assessment framework, they indicated that the four reviewers had reached 
a better level of consistency between the two STEM ability assessment instruments. In addition, based on the Chi- 
square test of goodness-of-fit, two research instruments of the STEM knowledge concept and the entrepreneurial 
scientific thinking scale are significant. The findings found that this STEM experience learning environment is 
feasible and effective and will help examine participants’ STEM abilities. However, the feasibility and effectiveness 
of evaluation results in the STEM practical learning environment mean that the two instruments are feasible and 
effective in the STEM ability assessment. The present study uses the scientific principles of the Royal Belgian siphon 
pots to create two summative STEM ability evaluation instruments. Their importance and use may increase the 
standard for STEM instruction and teaching effectiveness. 

On the top finding of that statement, the STEM knowledge concept and technological application of entre- 
preneurial scientific thinking in beverage preparation experience activities are rich in feasibility and effectiveness 
in the STEM ability assessment instruments development and evaluation framework. In other words, the Royal 
Belgian siphon pot integrates the scientific principles of vapour, gravity, pressure, and fire to construct a STEM 
learning environment in life science. Interdisciplinary learning will enrich learners’ vision and innovative value. It 
will help enhance learners’ feasibility and effectiveness and be meaningful for implementing the STEM practical 
learning environment. 


Limitations and Future Research 


This research highlights the role of educational technology in STEM practical learning environments. STEM 
practical learning enriches learners’ vision and innovative value and helps enhance their feasibility and effective- 
ness. That is meaningful in the STEM learning environment of life science. Above research results, we found that 
the learning environment is feasible and effective and will help examine participants’ STEM abilities. Therefore, 
according to the findings, the evaluation framework and STEM ability development tools are apparent when using 
the Royal Belgian siphon pot for making coffee. However, there are still some inference restrictions and suggestions 
on two aspects of teaching practice and follow-up research design to make the inference more cautious: 

1. Suggestions on teaching practice 
The teaching practice suggests that STEM education should integrate into the field of life science and 
technology. Let this pot not only be used in coffee brewing courses but can also expand to other bever- 
age brewing STEM courses, such as traditional tea brewing, scented tea brewing, and other life science 
curricula. Furthermore, the ease of use should increase in the operation and design of STEM practice 
learning environment. For example, to control and increase the extraction time until fully extracted to 
promote the taste, hoping to improve the learning efficiency of the experiencers. 

2. Suggestions for follow-up research design 
In future research design suggestions, the number of samples and the number of siphon pots discussed 
in this study are limited. Therefore, if we want to make broader inferences, we must proceed with cau- 
tion. We look forward to successfully increasing the number of effective samples and the number of 
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siphon pots in the future, enriching the STEM experience learning environment, and strengthening 
the depth and breadth of teaching practice to enhance learners’ STEM abilities. 
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Abstract. Redox reaction is a core 
chemical concept. However, its abstract 
nature makes it very difficult for students. 
Students’ conceptual structure reflects 
their mastery of concepts, which helps 
teachers implement targeted educational 
strategies. This study aimed to explore the 
conceptual structures of redox reaction 
held by students (grades 10 to 12) by 
employing MDS and HCA. A total of 606 
students participated, with 195 students 
in 10° grade, 202 in 11" grade, and 

209 in 12" grade. The results indicated 
that three-dimensional solutions were 
appropriate for the conceptual structures 
of 10° and 12" graders, while 11" graders 
demonstrated two-dimensional solutions. 
All students grouped the 15 concepts 
related to redox reaction into two large 
clusters: metrology and the redox reaction 
process. Moreover, both 10% and 12" 
graders further subdivided the 15 concepts 
into four subclusters: metrology, oxidation 
process, reduction process, and chemical 
reaction. Students’ conceptual structures 
were rational across all three grades. The 
conceptual structures of 10" and 12" 
graders were more refined than those of 
11* graders, and there was no significant 
difference between the conceptual 
structures of 10% and 12" graders. 11% 
graders learned about electrochemistry 
and tended to confuse concepts related 

to redox reaction with those related to 
electrochemistry. 
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Introduction 


Redox reaction is a core chemical concept taught throughout the upper- 
secondary chemistry curriculum. Knowledge of redox reaction is fundamental 
to much chemical knowledge, such as electrochemistry. However, due to 
the abstract nature of the concepts related to redox reaction and the limited 
exposure to them in daily life, it is often difficult for students to properly 
understand redox reaction (Brandriet & Bretz, 2014; Masykuri et al., 2019; 
Rosenthal & Sanger, 2012). 

Conceptual structure refers to the cognitive representation of a concept 
within an individual's mind, encompassing each of the concept’s components 
and the relationship among those components (Lin et al., 2022; Qian, 2008). 
A long-held assumption in cognitive and educational psychology holds that 
an individual should comprehend the interrelationships between concepts 
to be knowledgeable and aware of a certain domain (Goldsmith et al., 1991). 
In the process of understanding the interrelationships between concepts, 
the individual develops a conceptual structure of a certain domain in his or 
her mind, also called a knowledge structure (Hayes-Roth, 1977) or cognitive 
structure (Atabek-Yigit, 2018). Piaget’s theory of cognitive development 
posits that as an individual's knowledge expands, their cognitive structure 
undergoes ongoing development and enhancement (Supratman, 2013). In 
other words, students’ conceptual structure may change as existing knowl- 
edge establishes new connections with new knowledge (Casas-Garcia & 
Luengo-Gonzalez, 2013). 
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Based on the above analysis, it is essential to reveal the conceptual structures of upper-secondary school 
students at different grades concerning redox reaction. It can facilitate the accurate assessment of students 
conceptual mastery of redox reaction at various grades and provide valuable insights for educators to implement 
targeted teaching methods for students at different stages of learning. 

However, little existing research has focused on the conceptual structures of students regarding redox reac- 
tion (Chiang et al., 2014; Tang et al., 2022), and even less on the development of students’ conceptual structures 
regarding redox reaction (Chiang et al., 2014). 


’ 


Multidimensional Scaling and Hierarchical Cluster Analysis 


In the field of science education, various methods have been employed to understand the organization of 
concepts within the minds of individuals or groups, including concept maps (Burrows & Mooring, 2015; Edmond- 
son, 2005; He et al., 2023), word associations (Gulacar et al., 2015; Gulacar et al., 2022; Nakiboglu, 2008), pathfinder 
networks (Casas-Garcia & Luengo-Gonzalez, 2013; Fesel et al., 2015), factor analysis (Mai, Qian, Li et al., 2021; Tang 
et al., 2022), reaction time technique (Mai, Qian, Lan et al., 2021), multidimensional scaling (MDS) (Lin et al., 2022; 
Qian et al., 2023), and hierarchical cluster analysis (HCA) (Lin et al., 2022; Qian et al., 2023). 

Among these measures, MDS stands out as it's particularly advantageous for visualizing conceptual structure. 
MDS can create spatial maps that demonstrate the relative distances between objects, making it highly intuitive. It 
can also identify potential variables that determine multiple objects and depict them graphically in low-dimensional 
space, such as two-dimensional or three-dimensional space. The number of potential variables corresponds to 
the dimensions of the graph. In the spatial graph of potential variables, data points represent the objects under 
investigation, with closer proximity indicating greater similarity in terms of dimensional features (Luo & Zhao, 2005). 
Consequently, MDS enables researchers to quantitatively assess the similarity among concepts. 

However, the interpretation of MDS solutions is subjective (Hout et al., 2013). Different groupings of items 
will result in different potential variables. To address this issue, further analysis of the spatial coordinate values of 
the items using cluster analysis can be conducted (Lin et al., 2022). Cluster analysis allows categorization based on 
the distance among items. In the field of education and psychology, researchers commonly employ the K-Means 
cluster and HCA for data analysis. K-Means cluster is suitable when the number of clusters is known, while HCA is 
preferred in cases where the number of clusters is uncertain. In this study, HCA was utilized for analysis due to the 
uncertainty of the number of clusters of concepts in the students’ conceptual structures regarding redox reaction. 

Based on the previous analysis, MDS and HCA were employed in this study to explore and analyze students’ 
conceptual structures of redox reaction, respectively. 


Research Questions 


Prior research has been deficient in measuring and comparing changes in students’ conceptual structures 
across different grades. This study utilized MDS and HCA to explore and analyze students’ conceptual structures of 
redox reaction, respectively. The following were research questions that this study attempted to address. 

1. What are the conceptual structures of redox reaction held by 10", 11, and 12" graders, respectively? 
2. What are the similarities and differences among the conceptual structures of redox reaction in the 
minds of 10", 11, and 12‘ graders? 


Research Methodology 
General Background 


This study conducted a questionnaire survey on 195 Grade 10 students, 202 Grade 11 students, and 209 Grade 
12 students during the 2022-2023 academic year. This research was divided into three steps. 

Firstly, the participants categorized 15 concepts closely related to redox reaction based on similarity. Then, 
MDS and HCA were applied to explore and analyze students’ conceptual structures of redox reaction, respectively. 
Lastly, the similarities and differences among the conceptual structures of 10, 11%, and 12‘ graders regarding 
redox reaction were analyzed. 


165 


https://doi.org/10.33225/jbse/24.23.151 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


STUDENTS’ CONCEPTUAL STRUCTURES REGARDING REDOX REACTION: COMBINING ISSN 1648-3898 | /print 
MULTIDIMENSIONAL SCALING AND HIERARCHICAL CLUSTER ANALYSIS APPROACHES 
(PP. 151-174) ISSN 2538-71 38 /Online/ 


Participants 


The subjects of the study were students randomly selected from two upper-secondary schools with different 
academic levels in Guangzhou, Guangdong Province, China. Participants’ demographic information is displayed 
in Table 1. 

Before participating in the survey, all students had learned the knowledge of redox reaction in the compulsory 
curriculum of chemistry. Furthermore, 11‘ graders had completed the knowledge of electrochemistry, while 12" 
graders had finished all upper-secondary school chemistry knowledge and a round of comprehensive review in 
preparation for the college entrance exam. All subjects were informed in advance of the purpose of the test and 
participated voluntarily. 


Table 1 
Demographic Characteristics of Participants 


Age Gender 
Students N Recovery rate / % 
M SD Male Female 
10" graders 195 84.4 14.98 23 101 94 
11" graders 202 90.6 16.02 28 122 80 
12" graders 209 91.3 17.99 32 118 91 


Instrument and Procedures 


The instrument was a card-sorting questionnaire involving 15 items (as shown in Table 2). In our previous study, 
we identified 15 concepts that are critical to students’ understanding of redox reaction through content analysis, 
questionnaire survey, and interview (Tang et al., 2022). Given the objectivity, credibility, and representativeness 
of the results, these 15 concepts were chosen as items that are closely related to redox reaction in this study. The 
items in the questionnaire were disorganized, with no indication of relevance or classification. 

The card-sorting questionnaire was sent to the students (grades 10 to 12), and the 15 items were freely classi- 
fied according to their thoughts. There was no right or wrong classification, and the basis and number of categories 
for classification were determined by individuals. Each item was individually classified into one category without 
being omitted or repeatedly selected. Participants should complete the questionnaire within 20 minutes, and 
discussion during the process was forbidden. 


Table 2 
15 Items Closely Related to Redox Reaction 


Number Item Number Item Number Item 
1 Conservation of gain and loss electrons 6 Oxidation numbers 11 Reduced 
2 Electron transfer 7 Oxidation state changes 12 Reduction 
3 Oxidizing ability 8 Reducing agent 13 Oxidized 
4 Oxidizing agent 9 Oxidation 14 Oxidation product 
5 Number of gain and loss electrons 10 Reducing ability 15 Reduction product 


Note: The item “conservation of gain and loss electrons” refers to the principle that the number of electrons gained is equal to 
the number of electrons lost in a redox reaction (Tang et al., 2022). 
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Data Analysis 


The data analysis consisted of three steps. Firstly, the data from students at each grade were used to gener- 
ate a dissimilarity matrix for the corresponding grade level. A 15x15 dissimilarity matrix was created by assigning 
a score of 1 to item pairs in different groups and a score of 0 to item pairs in the same category (Qian et al., 2023). 
Thus, three dissimilarity matrices were obtained. 

Then, three dissimilarity matrices were analyzed via the MDS program of SPSS 23.0 to derive the conceptual 
structures of redox reaction for 10", 11", and 12 graders, respectively. The Euclidean squared distance was used 
to calculate the distance between items. 

Finally, HCA was conducted to analyze students’ conceptual structures of redox reaction by Ward’s method. 
The data used by HCA were the squared Euclidean distance between items. 


Research Results 
The Conceptual Structures of Redox Reaction Held by Students 


Researchers used Stress and RSQ to evaluate the quality of the data fit while employing MDS to analyze data. 
Stress indicates the fitness of the statistical results to the original data; the smaller, the better. Lower stress values 
(below .10) indicate a better fit (Roy, 2020), as shown in Table 3. RSQ indicates the square of the regression coef- 
ficient, reflecting the proportion of the variance of the original data explained by the low-dimensional space; the 
closer the RSQ is to 1, the better. 

Table 4 shows the values of Stress and RSQ in two- and three-dimensional solutions for students (grades 
10 to 12). Three-, two- and three-dimensional solutions were appropriate for students in Grades 10, 11, and 12, 
respectively, owing to the stress and RSQ. In other words, the conceptual structures of redox reaction in the minds 
of 10", 11°, and 12" graders were three-, two-, and three-dimensional, respectively (shown in Figure 1). 

It is worth noting that both the two- and three-dimensional solutions demonstrated a good fit with the data 
for 11" graders. However, upon a thorough analysis of the images from the two-dimensional and three-dimensional 
solutions, it was discovered that the 15 items were clustered in two locations, with each cluster being closely 
grouped. Furthermore, the two-dimensional solution allowed for a more intuitive representation of the aggrega- 
tion among items. Therefore, the two-dimensional solution was accepted to serve as the conceptual structure of 
redox reaction for 11 graders. 

To represent more clearly and intuitively the clustering among items in the three-dimensional solution im- 
age, Figures 2 and 3 present 10" and 12" graders’ two-dimensional mapping images of the three-dimensional 
solutions (Lin et al., 2022). 


Table 3 
Stress Values and Corresponding Goodness of Fit 


Stress Goodness of Fit 
.200 Poor 
.100 Fair 
.050 Good 
025 Excellent 
.000 Perfect 
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Table 4 


Stress and RSQ Values Determined by MDS 


Students 


10th graders 
11th graders 


12th graders 


Figure 1 


Stress 


Model Fit 


Two Dimensions 


RSQ 


947 
.999 


943 


Conceptual Structures of Students in Grades 10, 11, and 12 


Dimension 2 


“4 dg TF yps oft 
, 2 or 
Dimension 4 pier 


(a) 10" graders 


Figure 2 


Dimension 2 


Dimension 1 


(b) 11" graders 


Two Dimensional Projections of the Three-dimensional Solution of 10" Graders 
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Three Dimensions 


Stress RSQ 
.087 956 
.030 998 
.092 958 


Dimension 2 


08. 
ist yon? 
7 on 
Dimension 1 pimens® 


(c) 12" graders 


Dimer 
a 


Figure 3 


Dime! 


(a) D1 and D2 (b) D1 and D3 (c) D2 and D3 
Note: “D" refers to dimension. 
Two Dimensional Projections of the Three-dimensional Solution of 12" Graders 
(a) D1 and D2 (b) D1 and D3 (c) D2 and D3 


Note: “D" refers to dimension. 
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A three-dimensional solution was appropriate for the conceptual structure of 10‘ graders. The names of the 
three dimensions were as follows, based on the shared meaning of items in each dimension. 

First, D1 was named “metrology/redox reaction process” The left side of D1 included five items related mostly 
to metrology, while ten items on the right were related to the redox reaction process, as demonstrated in Figure 
2(a). Second, D2 was named “reduction process/oxidation mechanism”. Five items related to the reduction process 
were observed on the opposing side of D2, while ten items associated with the oxidation mechanism were observed 
on the positive side of D2. Finally, D3 was named “redox reaction mechanism/chemical reaction”. The lower side of 
D3 gathered items related to the redox reaction mechanism, while the upper side gathered items “oxidation” and 
“reduction’, both referred to chemical reactions, as shown in Figure 2(b). 

A two-dimensional solution was appropriate for the conceptual structure of 11" graders. In the two-dimen- 
sional solution image, quadrants were numbered from one to four (Q1, Q2, Q3, and Q4) ina counterclockwise order 
setting up with the top-right quadrant (Yilmaz & Kapkin, 2021). As illustrated in Figure 1(b), 15 items were mainly 
clustered in Q2 and Q4. Five items in Q2 related to metrology, which was consistent with the items to the left of D1 
in the three-dimensional image of 10‘ graders. Ten items in Q4 were related to the redox reaction process, which 
was consistent with the items to the right of D1 in the three-dimensional image of 10" graders. 

A three-dimensional solution was appropriate for the conceptual structure of 12" graders. A careful compari- 
son of Figures 2 and 3 reveals that in terms of the distribution of concepts in each dimension, 12" and 10° graders 
had extremely similar conceptual structures. As a result, each dimension in the three-dimensional solution of 12" 
graders was labeled as follows. The D1, D2, and D3 were named “metrology/redox reaction process’, “reduction 
process/oxidation mechanism’, and “redox reaction mechanism/chemical reaction’, respectively. 


Conceptual Clustering in the Conceptual Structures of Redox Reaction Held by Students 
The dendrograms obtained by HCA are shown in Figure 4. 15 items were divided into two large clusters in- 
cluding four sub-clusters by 10 and 12" graders, while they were split into two clusters only by 11" graders. Table 


5 displays the classifications examined by HCA of students in different grades. 


Figure 4 
The Dendrograms Obtained by HCA of 10", 11, and 12" graders 


(a) 10" graders (b) 11" graders (c) 12" graders 


Table 5 
The Classifications Obtained by HCA of 10", 11, and 12" graders 


Students Large clusters Sub-clusters 
Metrology (1) Metrology 
10" graders 
Redox reaction process (2) Oxidation process; (3) Reduction process; (4) Chemical reaction 
Metrology - 
11th graders 
Redox reaction process - 
Metrology (1) Metrology 
12" graders 
Redox reaction process (2) Oxidation process; (3) Reduction process; (4) Chemical reaction 
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As demonstrated in Table 5, all three grade students divided the 15 items into two large clusters. One cluster 
included five items regarding metrology, and the other comprised ten items concerning the redox reaction process. 
10° and 12" graders continued to subdivide the cluster “redox reaction process” into three subclusters, in 


wu 


contrast to 11" grades. The three subclusters were “oxidation process’, “reduction process’, and “chemical reaction”. 


Thus, 10° and 12" graders split the 15 items into four sub-clusters: “metrology”, “oxidation process’, “reduction 
process’, and “chemical reaction”. 


Discussion 


In this study, MDS and HCA were utilized to explore and analyze the conceptual structures of students (grades 
10 to 12) regarding redox reaction, respectively. The similarities and differences among the conceptual structures 
of redox reaction in the minds of 10", 11", and 12" graders were analyzed. 


Similarity of Conceptual Structures Held by Students in Three Grades 


Although the organizations of the conceptual structures regarding redox reaction were not the same for 
the students in all three grades, their conceptual structures were all rational. To begin with, students categorized 
15 concepts relevant to redox reaction correctly and with clear boundaries. The MDS results showed that the 15 
concepts were distinctly and unambiguously categorized by students, with a high degree of concentration within 
each category. It indicated that students were able to clarify the interrelationships of 15 concepts relevant to redox 
reaction. 

Second, there was a good scientific connection among the concepts clustered into a category. The HCA results 
revealed that the students in all three grades grouped the 15 concepts into the same two large clusters. The names 
and specific distribution of the concepts are presented in Table 6. 


Table 6 
The Concepts Included in Two Large Clusters 


Large cluster Name Concepts 
Electron transfer, Oxidation numbers, Oxidation state changes, Number of gain and loss 
1 Metrology ; : 
electrons, Conservation of gain and loss electrons 
9 Redox reaction process Oxidizing agent, Oxidizing ability, Reduced, Reduction product, Reducing agent, Reducing 


ability, Oxidized, Oxidation product, Oxidation, Reduction 


In large cluster 1, the essence of the redox reaction is electron transfer, involving electron gain and loss. The 
number of electrons lost equals the number of electrons gained, which is what the concept“conservation of gain and 
loss electrons” means (Tang et al., 2022). Due to electron transfer, the oxidation numbers of the element will change. 

In large cluster 2, the oxidizing agent has the oxidizing ability and is reduced during a reduction to produce 
reduction products. Meanwhile, a reducing agent has a reducing ability and is oxidized in the process of oxidation 
to produce oxidation products. 

In conclusion, the 15 concepts connected to redox reaction were categorized and then stored in students 
minds with clear boundaries between categories. Meanwhile, concepts in each category had a scientific connec- 
tion to each other. That is, students in all three grades were able to comprehend the meaning of the 15 concepts 
and their interrelationships. 


o 


Differences of Conceptual Structures Held by Students in Three Grades 


The conceptual structures of 10" and 12" graders were more refined than those of 11" graders, while there 
was no significant difference between the conceptual structures of 10" and 12" graders. 
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As shown in Figure 4, 11" graders clustered the 15 concepts related to redox reaction into two categories, while 
both 10* and 12" graders further subdivided the 15 concepts into four categories, each with chemical significance 
and more refinement. The specific concept distribution and names of the four subclusters are shown in Table 7. 

Comparison of Tables 6 and 7 shows that 10" and 12‘ graders continued to subdivide the cluster “redox 
reaction process” into three subclusters. The three subclusters were “oxidation process’, “reduction process”, and 
“chemical reaction”. A redox reaction consists of two half-reactions, namely an oxidation and a reduction. The redox 
reaction process encompasses both the oxidation process and the reduction process. Consequently, additional 
categorization by 10‘ and 12" graders was justified, suggesting that 10" and 12" graders exhibit a more optimal 
cognitive processing of the 15 concepts. 

11 graders had just completed the knowledge of electrochemistry at the time of the survey. In an electro- 
chemical device, oxidation and reduction occur at different sites and are usually analyzed separately. For example, 
oxidation occurs at the anode and reduction at the cathode. Therefore, 11" graders were expected to be able to 
further refine their categorization of the cluster “redox reaction process”. However, it is surprising that 11" graders 
did not subdivide the cluster “redox reaction process” into “oxidation process” and “reduction process”. 

Interviews with upper-secondary chemistry teachers and 11" graders revealed the following two causes. Firstly, 
11" graders had forgotten their understanding of redox reaction to a larger extent. In China, upper-secondary school 
students are studying the knowledge of redox reaction in their first year of chemistry curriculum. At the time of the 
survey, it had been a long time since 11" graders had studied redox reaction in depth. As a result, students forgot 
the meaning and relationship of concepts related to redox reaction to a larger extent. 

Secondly, the knowledge of electrochemistry interfered with the conceptual structure of 11 graders regarding 
redox reaction. Many of the concepts related to electrochemistry are very abstract, such as anode, cathode, reduc- 
tion potential, and so on. Students need to use the knowledge of redox reaction to understand these concepts. 
For example, the site of the oxidation reaction is the anode. Thus, students tended to confuse concepts related to 
redox reaction with those related to electrochemistry and then affected the conceptual structure of redox reac- 
tion for 11" graders. 

In an earlier study, Chiang et al. (2014) explored the conceptions of Taiwanese, China upper-secondary school 
students (grades 10 to 12) regarding redox reactions via a two-tier test. It was found that some 11" graders under- 
stood redox reactions better than the 12‘ graders. The researchers attributed this finding to the fact that the 12" 
graders studied electrochemistry, which led to conceptual confusion between redox and electrochemical theories. 
This result is consistent with our results for the following reason. 

In the Chinese mainland, students finish learning electrochemistry in their sophomore year of upper-secondary 
school. Thus, 11 graders are disturbed by their electrochemical knowledge in understanding redox reactions 
compared to 10° graders. Although 12" graders also learned about the knowledge of electrochemistry, they were 
less likely to be disturbed by the knowledge of electrochemistry in their understanding of redox reactions. At the 
time of the survey, 12" graders had completed all upper-secondary school chemistry knowledge and a round of 
review in preparation for the college entrance exam. Consequently, they developed a stronger, more refined, and 
systematic understanding of the core concepts of redox reaction as they solved problems in complex systems. 

Notably, our work is still significant even if Chiang et al. (2014) studied the knowledge structures of redox 
reactions that students (grades 10 to 12) held. First, the students in the two studies had different knowledge 
foundations. While students in the Chinese mainland learn about electrochemistry in their sophomore year of 
upper-secondary school and spend their senior year of upper-secondary school doing multiple rounds of review 
for the college entrance examination, students in Taiwan, China, study electrochemistry in their senior year of high 
school. Students’ comprehension of redox reactions is affected by both their knowledge of electrochemistry and 
the several rounds of knowledge reviews. 

Second, this study is the initial attempt to utilize MDS and HCA to explore and analyze the conceptual structure 
of redox reactions in students’ minds. The conceptual structure obtained in this study gives access to the spatial 
representation of the conceptual structure as well as the conceptual clustering within it (Lin et al., 2022; Qian et al., 
2023), providing insight into the degree of refinement of students’ understanding of redox reactions. 
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Table 7 
The Concepts Included in Each Subcluster 


Subcluster Name Concepts 


Electron transfer, Oxidation numbers, Oxidation state changes, Number of gain and loss 


‘ Metology electrons, Conservation of gain and loss electrons 

2 Oxidation process Oxidizing agent, Oxidizing ability, Reduced, Reduction product 
3 Reduction process Reducing agent, Reducing ability, Oxidized, Oxidation product 
4 Chemical reaction Oxidation, Reduction 


Conclusions and Implications 


This study utilized MDS and HCA to explore and analyze students’ conceptual structures of redox reac- 
tion, respectively. In the minds of 10", 11", and 12" graders, the conceptual structures of redox reaction were 
three-, two-, and three-dimensional, respectively. The 15 concepts related to redox reaction were grouped into 
two large clusters by students in all three grades. Furthermore, both 10% and 12" graders subdivided the 15 
concepts into four subclusters. 

Students’ conceptual structures were rational across all three grades. The conceptual structures of 10% and 
12" graders were more scientific and rational than those of 11" graders, and there was no significant difference 
between the conceptual structures of 10" and 12" graders. 

The preceding conclusions offer two insights. First, teachers can tailor instruction to students based on the 
conceptual structures of students at different grades. For instance, teachers should assist students in reviewing 
the definitions of concepts relevant to redox reaction and their interrelationships before instructing 11 grad- 
ers on electrochemistry. It will help 11 graders develop a scientifically sound conceptual structure for redox 
reactions and will also contribute to their understanding of electrochemistry. 

Second, subsequent researchers can examine students’ conceptual structures across disciplines. Different 
disciplines may cover the same knowledge. For example, the knowledge of molecular structure is found in both 
upper-secondary school chemistry and biology curriculums. The concepts closely linked to this knowledge 
are distributed in different disciplines, so the corresponding conceptual structures in students’ minds may 
integrate knowledge from multiple disciplines. Researchers can select closely related concepts of a particular 
piece of knowledge from multiple related disciplines so that the resulting conceptual structure can be use- 
ful in adequately reflecting students’ understanding and organization of knowledge. It will support teachers 
in implementing effective teaching strategies that foster students’ integrative thinking and interdisciplinary 
problem-solving skills. 


Acknowledgments 


This research is supported by the project sponsored by MOE of P.R.C: Reforming the teaching paradigm of 
basic education in the new era: from “input-oriented” to “output-oriented”. 


Declaration of Interest 
The authors declare no competing interest. 


References 


Atabek-Yigit, E. (2018). Can cognitive structure outcomes reveal cognitive styles? A study on the relationship between cognitive 
styles and cognitive structure outcomes on the subject of chemical kinetics. Chemistry Education Research and Practice, 
19(3), 746-754. https://doi.org/10.1039/c8rp00018b 


172 


https://doi.org/10.33225/jbse/24.23.151 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


STUDENTS’ CONCEPTUAL STRUCTURES REGARDING REDOX REACTION: COMBINING ISSN 1648-3898 = /erint, 
MULTIDIMENSIONAL SCALING AND HIERARCHICAL CLUSTER ANALYSIS APPROACHES 
rap iconeie ISSN 2538-7138 sontine/ 


Brandriet, A. R., & Bretz, S. L. (2014). Measuring meta-ignorance through the lens of confidence: Examining students’ redox 
misconceptions about oxidation numbers, charge, and electron transfer. Chemistry Education Research and Practice, 
15(4), 729-746. https://doi.org/10.1039/C4RP001295J 

Burrows, N.L., & Mooring, S. R. (2015). Using concept mapping to uncover students’ knowledge structures of chemical bonding 
concepts. Chemistry Education Research and Practice, 16(1), 53-66. https://doi.org/10.1039/C4RP00180J 

Casas-Garcia, L. M., & Luengo-Gonzalez, R. (2013). The study of the pupil’s cognitive structure: the concept of angle. European 
Journal of Psychology of Education, 28, 373-398. https://doi.org/10.1007/s10212-012-0119-4 

Chiang, W.-W., Chiu, M.-H., Chung, S.-L., & Liu, C.-K. (2014). Survey of high school students’ understanding of oxidation-reduction 
reaction. Journal of Baltic Science Education, 13(5), 596-607. https://doi.org/10.33225/jbse/14.13.596 

Edmondson, K. M. (2005). Assessing science understanding through concept maps. In Mintzes, J. J., Wandersee, J. H., & 
Novak, J. D. (Eds.), Assessing science understanding: A human constructivist view (1st ed., pp. 15-40). Academic Press. 
https://doi.org/10.1016/B978-0-12-498365-6.X5000-8 

Fesel, S. S., Segers, E., Clariana, R. B., & Verhoeven, L. (2015). Quality of children’s knowledge representations in 
digital text comprehension: Evidence from pathfinder networks. Computers in Human Behavior, 48, 135-146. 
https://doi.org/10.1016/j.chb.2015.01.014 

Goldsmith, T. E., Johnson, P. J., & Acton, W. H. (1991). Assessing structural knowledge. Journal of Educational Psychology, 83(1), 
88-96. https://doi.org/10.1037//0022-0663.83.1.88 

Gulacar, O., Sinan, O., Bowman, C. R., & Yildirim, Y. (2015). Exploring the changes in students’ understanding of the scientific 
method using word associations. Research in Science Education, 45, 717-726. https://doi.org/10.1007/s11165-014-9443-9 

Gulacar, O., Vernoy, B., Tran, E., Wu, A., Huie, E. Z., Santos, E. V., Wadhwa, A., Sathe, R., & Milkey, A. (2022). Investigating differences 
in experts’ chemistry knowledge structures and comparing them to those of general chemistry students. Journal of 
Chemical Education, 99(8), 2950-2963. https://doi.org/10.1021/acs.jchemed.2c00251 

Hayes-Roth, B. (1977). Evolution of cognitive structures and processes. Psychological Review, 84(3), 260-278. 
https://doi.org/10.1037/0033-295X.84.3.260 

He, X., Fang, J., Cheng, H. N., Men, Q., & Li, Y. (2023). Investigating online learners’ knowledge structure patterns 
by concept maps: A clustering analysis approach. Education and Information Technologies, 28, 11401-11422. 
https://doi.org/10.1007/s10639-023-11633-8 

Hout, M. C., Papesh, M. H., & Goldinger, S. D. (2013). Multidimensional scaling. Wiley Interdisciplinary Reviews Cognitive Science, 
4(1), 93-103. https://doi.org/10.1002/wcs.1203 

Lin, H., Qian, Y., Wen, J., & Mai, Y. (2022). Utilizing multidimensional scaling to represent the conceptual structures of atomic 
structure in upper-secondary school teachers and students. Journal of Baltic Science Education, 21(3), 481-494. 
https://doi.org/10.33225/jbse/22.21.481 

Luo, W., & Zhao, S. (2005). Duoweichidufa jiqi zai xinlixue lingyu zhong de yingyong [Multidimensional scaling approach and its 
application in the field of psychology]. China Examinations, (4), 4. https://doi.org/10.19360/j.cnki.1 1-3303/g4.2005.04.009 

Mai, Y., Qian, Y., Lan, H., & Li, L. (2021). Students’ concept organisation regarding chemical equilibrium in upper- 
secondary education: Based on reaction time technique. Journal of Baltic Science Education, 20(3), 443-455. 
https://doi.org/10.33225/jbse/21.20.443 

Mai, Y., Qian, Y., Li, L., & Lan, H. (2021). The conceptual structure of chemical equilibrium in upper-secondary school students: 
Evidence from factor analysis. Journal of Baltic Science Education, 20(1), 80-92. https://doi.org/10.33225/jbse/21.20.80 

Masykuri, M., Afifa, F.N., & Ashadi, A. (2019). Students’ misconceptions on basic concept of redox reaction. International Conference 
on Science, Mathematics, Environment, and Education, AIP Publishing, 2194(1). https://doi.org/10.1063/1.5139794 

Nakiboglu, C. (2008). Using word associations for assessing non major science students’ knowledge structure before and after 
general chemistry instruction: the case of atomic structure. Chemistry Education Research and Practice, 9(4), 309-322. 
https://doi.org/10.1039/b818466f 

Qian, Y. (2008). Gaozhong shisheng huaxue xueke guanjianci gainian jiegou de yanjiu [A study on high school teachers and 
students’ concept structure of keywords in chemistry] [Unpublished doctoral dissertation]. South China Normal University. 

Qian, Y., Wang, Y., Wen, J., Wu, S., & Zhang, J. (2023). One hundred core concepts in chemistry and upper-secondary 
school teachers’ and students’ chemistry conceptual structures. Journal of Baltic Science Education, 22(3), 493-505. 
https://doi.org/10.33225/jbse/23.22.493 

Rosenthal, D. P., & Sanger, M. J. (2012). Student misinterpretations and misconceptions based on their explanations of two 
computer animations of varying complexity depicting the same oxidation-reduction reaction. Chemistry Education 
Research and Practice, 13(4), 471-483. https://doi.org/10.1039/c2rp20048a 

Roy, A. (2020). Studying positioning and repositioning of brands using multidimensional scaling. Journal of Education for 
Business, 95(1), 53-58. https://doi.org/10.1080/08832323.2019.1599795 

Supratman, A. M. (2013). Piaget’s theory in the development of creative thinking. Research in Mathematical Education, 17(4), 
291-307. https://doi.org/10.7468/jksmed.2013.17.4.291 

Tang, W., Zhu, X., & Qian, Y. (2022). Applying factor analysis for assessing knowledge structure of students in grade 10: the 
subject of redox reaction. Journal of Baltic Science Education, 21(4), 680-693. https://doi.org/10.33225/jbse/22.21.680 


173 


https://doi.org/10.33225/jbse/24.23.151 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


STUDENTS’ CONCEPTUAL STRUCTURES REGARDING REDOX REACTION: COMBINING ISSN 1648-3898 | /printy 
MULTIDIMENSIONAL SCALING AND HIERARCHICAL CLUSTER ANALYSIS APPROACHES 
(PP. 151-174) ISSN 2538-71 38 /Online/ 


Yilmaz, T., & Kapkin, E. (2021). Mapping the intervention of sustainable design studio using multidimensional 
scaling and pathfinder networks. /nternational Journal of Sustainability in Higher Education, 22(2), 308-327. 
https://doi.org/10.1108/ijshe-04-2020-0125 


Received: November 06, 2023 Revised: December 11, 2023 Accepted: January 08, 2024 


Cite as: Tang, W., Qian, Y., Wang, H., Wen, J., Huang, J., Zhu, X., & Wang, Y. (2024). Students’ conceptual structures regarding 
redox reaction: Combining multidimensional scaling and hierarchical cluster analysis approaches. Journal of Baltic Science 
Education, 23(1), 164-174. https://doi.org/10.33225/jbse/24.23.164 


Wenxiu Tang PhD student, College of Teacher Education, South China Normal 
University, No.55, West of Zhongshan Avenue, Tianhe District, 
Guangzhou, P. R. China. 
E-mail: tangwenxiu@m.scnu.edu.cn 
ORCID: https://orcid.org/0000-0002-3326-4171 


Yangyi Qian PhD, Professor, School of chemistry, South China Normal University, 
(Corresponding author) No.378, West Waihuan Street, Higher Education Mega Center, Panyu 
District, Guangzhou, P. R. China. 
E-mail: qianyy@scnu.edu.cn 
ORCID: https://orcid.org/0000-0002-4587-6298 


Hong Wang PhD, Professor, College of Teacher Education, South China Normal 
University, No.55, West of Zhongshan Avenue, Tianhe District, 
Guangzhou, P. R. China. 
E-mail: 884775366@qq.com 
ORCID: https://orcid.org/0009-0008-7026-9052 


Jinju Wen Postgraduate Student, School of chemistry, South China Normal 
University, No. 378, West Waihuan Street, Higher Education Mega Center, 
Panyu District, Guangzhou, P.R.China. 
E-mail: 2667222624@qq.com 
ORCID: https://orcid.org/0000-0001-9784-5840 


Jiayi Huang Postgraduate Student, School of chemistry, South China Normal 
University, No.378, West Waihuan Street, Higher Education Mega Center, 
Panyu District, Guangzhou, P.R.China. 
E-mail: 2622885795@qq.com 
ORCID: https://orcid.org/0000-0001-7117-2916 


Xintong Zhu Teacher, Dongguan Middle School Songshan Lake School, No.1 
Hongmian Road, Songshan Lake Science and Technology Industrial Park, 
Dongguan, P. R. China. 
E-mail: 390779509@qq.com 
ORCID: https://orcid.org/0000-0002-0763-719X 


Yangqian Wang PhD Student, School of Psychology, South China Normal University, 
No.55, West of Zhongshan Avenue, Tianhe District, Guangzhou, P. R. 
China. 
E-mail: wangyangqian718@163.com 
ORCID: https://orcid.org/0000-0003-0940-8195 


174 


https://doi.org/10.33225/jbse/24.23.151 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


ISSN 1648-3898 print 
ISSN 2538-7138 /oniine/ 


INFORMATION 
FOR CONTRIBUTORS 


EDITORIAL AND REVIEW PROCEDURES 


Journal of Baltic Science Education (JBSE) publishes original scientific research articles in the field of Natural Science Education and 
related areas for all educational levels in the Baltic and other countries. It is possible to publish special (thematic) issues of JBSE. The 
papers should be submitted and will be published in English. JBSE will promote to establish contacts between researchers and practical 
educators both in the Baltic countries and countries around. 

The authors of the manuscripts are responsible for the scientific content and novelty of the research materials. Articles, published 
before in other international journals or papers’ collections will not be accepted for publication in JBSE. 

As a publication that represents a variety of cross-disciplinary interests, both theoretical and practical, the JBSE invites manuscripts on 
a wide range of topics, especially in the following areas: 


+ Didactics of natural sciences. + Philosophical, political, economical and social aspects 
+ Theory and practice in natural science teacher of natural science education. 
education. + The supplementary natural science education. 
+ Integrated natural science education. + ICT in natural science education. 
+ Natural science and technological literacy. + The standardisation of natural science education etc. 


+ General and professional natural science education. 


MANUSCRIPTS GUIDELINES 


The structure of the research paper presented to the Journal of Baltic Science Education should be as follows: abstract - short report of the 
investigation; introduction inc. aim and subject of the research; research methodologies and methods; results of the research incl. discussion; 
conclusions; list of references in APA style (7th Ed.). 

The papers should be submitted in English. If English is a second language for the author, please consider having the manuscript 
proof read and edited before submitting. 


The text must be elaborated in Word for Windows, using 12 point Times New Roman letters. An article should not exceed 7-10 A4 pages, 
included figures, tables and bibliography. Publishing of longer articles should be negotiated separately. Texts margins: top and bottom 20mm, 
left -25mm, right - 20mm. The title: capital letters, 14pt, bold; space between the title and the author's name is one line interval. Author's name 
and surname: small letters, 12pt, bold. Under the name, institution: 11 pt, italics; space between the title and the text: 1 line interval. Abstract 
- about 100-150 words - precedes the text. The text: 12pt Single or Auto spacing, in one column. Key words: no more than five words. The 
language must be clear and accurate. The authors have to present the results, propositions and conclusions in a form that can suit scientists 
from different countries. 


Titles of the tables and figures: 11 pt, small letters. Space between figures or tables and the text: 1 line interval. Introduction, titles of 
chapters and subchapters: 12pt, bold, small letters. Numbers: Arabic, subchapters numbered by two figures (1.1, 1.2, etc.). Figures, tables and 
captions should be inserted within the manuscript at their appropriate locations. Diagrams and graphs should be provided as finished black 
and white line artwork or electronic images. When there are a number of illustrations, the author should endeavour to reduce the amount of 
text to accommodate the illustrations in the limited space available for any article. 


References in the text should be presented in brackets (Knox, 1988; Martin, 1995). If necessary, the page can be indicated: (Martin, 
1995, p.48). The list of references should be presented after the text. The Words List of References: 11pt, bold, small letters. The references 
should be listed in full at the end of the paper in the following standard form: 

For books: Saxe, G.B. (1991). Cultural and Cognitive Development: Studies in Mathematical Understanding. Hillsdale, NJ: Erlbaum. 

For articles: Bekerian, D.A. (1993). In Search of the Typical Eyewitness. American Psychologist, 48, 574-576. 

For chapters within books; Bjork, R.A. (1989). Retrieval Inhibition as an Adaptive Mechanism in Human Memory. In: H.L. Roediger Ill & F.I.M. 

Craik (Eds.), Varieties of Memory & Consciousness (pp. 309-330). Hillsdale, NJ: Erlbaum. 


On a separate page, author - related data should be presented in English: name, surname, degree and academic title, institution, full 
correspondence address in the clearest and most complete form /ordinary post and e-mail addresses /, position (to ensure anonymity in the 
review process). The author (authors) should confirm in writing, that the manuscript has not been published in other journal or 
handed over (transferred) to other journal for publication. 


EDITORIAL AND REVIEW PROCEDURES 


Manuscripts will be sent anonymously to reviewers with expertise in the appropriate area. All manuscripts will be rewieved by two 
experts before JBSE’s accept them for publication. This process usually takes about two months. The journal co-editors will make minor 
editorial changes; major changes will be made by the author(s) prior to publication if necessary. JBSE’s redaction will sent to author(s) 
only one correcture which must be sent back within 2 weeks. JBSE will not review submissions previously published elsewhere through 
print or electronic medium. 


Manuscripts submitted to the JBSE cannot be returned to authors. Authors should be sure to keep a copy for themselves. Authors’ 
signatures should be at the end of the paper and its second checked proofs. 


Manuscripts, editorial correspondence (and other correspondence for subscription and exchange), and any questions should be 
sent to editor-in-chief or to regional redactors. 


Journal's requirements for the authors are available online: http://www.scientiasocialis.It/jbse/files/JBSE_requirements_2019.pdf 


175 


Journal of Baltic Science Education, Vol. 23, No. 1, 2024 


ISSN 1648-3898 /Print/, ISSN 2538-7138 /Online/ 


Compiled by: Vincentas Lamanauskas 
Linguistic Editor: Ilona Ratkeviciené 
Cover design by: Jurgina Jankauskiené 

Layout design by: Linas Janonis 


28 April 2024. Publishing in Quires 10.5. Edition 90 


Publisher Scientia Socialis Ltd., 
Donelaicio Street 29, LT-78115 Siauliai, Lithuania 
E-mail: scientia@scientiasocialis.It 
Phone: +370 687 95668 
http://www.scientiasocialis.It 


Printing Siauliai printing house 
9A P. LukSio Street 
LT-76207 Siauliai, Lithuania 
Phone: +370 41 500 333. 
Fax: +370 41 500 336 
E-mail: info@dailu.lt 
https://siauliuspaustuve.|t/ 


Journal 
E of Baltic Science 


ducation 


PARENTS’ VIEWS ON THE USE OF AI-BASED 
CHATBOTS SUCH AS ChatGPT IN HIGH 
SCHOOL (STEM) EDUCATION 

Markus Sebastian Feser 


LIVE-STREAMING PERFORMANCE IN 
INQUIRY-BASED SCIENCE LEARNING WITH 
ACTION: TEACHERS’ PERSPECTIVES 
Jon-Chao Hong, Huei-Ying Ho, Ming-Chou 
Liu, Kai-Hsin Tai 


EFFECTS OF DIGITAL GAME-BASED 
LEARNING IN STEM EDUCATION ON 
STUDENTS’ MOTIVATION: A SYSTEMATIC 
LITERATURE REVIEW 

Jelena Ili¢, Mirjana Ivanovic, Aleksandra 
Klasnja-Milicevic 


A META-ANALYTIC EXAMINATION OF THE 
EFFECT OF INDIVIDUAL AND SCHOOL- 
AVERAGE ACHIEVEMENT ON STUDENTS’ 
SELF-CONCEPT IN SCIENCE 

Nurcan Kahraman 


DIFFERENCES IN THE WISHES OF 
STUDENTS, TEACHERS, AND PARENTS ON 
INTEGRATION OF SMARTPHONES AND 
TABLETS IN BIOLOGY LESSONS 

Vida Lang, Andrej Sorgo 


DEVELOPING A KNOWLEDGE MAP FOR THE 
EARLY CHILDHOOD STEAM EDUCATION: 

A VISUAL ANALYSIS USING CITESPACE AND 
HISTCITE 

Yumeng Li, Corrienna Abdul Talib 


KNOWLEDGE AND ATTITUDES TOWARD 
BIOTECHNOLOGY IN STEM EDUCATION 

AS AN INDICATOR OF SCIENTIFIC LITERACY 
Ruo-Yang Li, Yu-Yang Ma 


ISSN 1648-3898 /Print/ 
ISSN 2538-7138 /Online/ 


THE IMPACT OF THE OK4R MOBILE 
PLATFORM ON VOCATIONAL STUDENTS’ 
IMPROVEMENT OF SCIENCE READING 
COMPREHENSION AND LOGICAL THINKING 
Pao-Yin Lin, Pei-yu Chen, Yuan-Chen Liu, 
Hsiao-Ching Yang 


THE EFFECTS OF STUDENTS’ STANDPOINTS 
IN ARGUMENTATION REGARDING SOCIO- 
SCIENTIFIC ISSUES 

Yu-Ren Lin, Tzu-Ting Wei 


STUDENTS’ REFLECTIONS ON THEIR 
SCIENTIST- OR ENGINEER-LIKE PRACTICES 
IN STEM PROJECT-BASED LEARNING 

Tian Luo, Jiayue Zhao, Winnie Wing Mui 
So, Wencong Zhan 


ENVIRONMENTAL ISSUES ON TIKTOK: 
TOPICS AND CLAIMS OF MISLEADING 
INFORMATION 

Brenda Braga Pereira, Sangwoo Ha 


THE FEASIBILITY AND EFFECTIVENESS OF 
LIFE SCIENCE IN STEM PRACTICAL 
LEARNING ENVIRONMENT 

King-Dow Su 


STUDENTS’ CONCEPTUAL STRUCTURES 
REGARDING REDOX REACTION: COMBINING 
MULTIDIMENSIONAL SCALING AND 
HIERARCHICAL CLUSTER ANALYSIS 
APPROACHES 

Wenxiu Tang, Yangyi Qian, Hong Wang, 
Jinju Wen, Jiayi Huang, Xintong Zhu, 
Yangqian Wang 


